-- Logs begin at Wed 2022-09-28 11:33:46 UTC, end at Wed 2022-09-28 13:03:49 UTC. -- Sep 28 12:00:12.693278 dv-cinder-859193-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 28 12:00:12.701886 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 28 12:00:12.703882 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: *** Starting uWSGI 2.0.18-debian (64bit) on [Wed Sep 28 12:00:12 2022] *** Sep 28 12:00:12.703882 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 28 12:00:12.703882 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 28 12:00:12.703882 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: nodename: dv-cinder-859193-2-iscsi Sep 28 12:00:12.703882 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: machine: x86_64 Sep 28 12:00:12.703882 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: clock source: unix Sep 28 12:00:12.703882 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: pcre jit disabled Sep 28 12:00:12.703882 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: detected number of CPU cores: 4 Sep 28 12:00:12.703882 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: current working directory: / Sep 28 12:00:12.703882 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: detected binary path: /usr/bin/uwsgi-core Sep 28 12:00:12.703882 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: your processes number limit is 47680 Sep 28 12:00:12.703882 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: your memory page size is 4096 bytes Sep 28 12:00:12.703882 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: detected max file descriptor number: 2048 Sep 28 12:00:12.703882 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: lock engine: pthread robust mutexes Sep 28 12:00:12.704629 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: thunder lock: enabled Sep 28 12:00:12.704629 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 28 12:00:12.704629 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 28 12:00:12.743978 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: Python main interpreter initialized at 0x55ce0b7f09c0 Sep 28 12:00:12.743978 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: python threads support enabled Sep 28 12:00:12.743978 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: your server socket listen backlog is limited to 100 connections Sep 28 12:00:12.743978 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: your mercy for graceful operations on workers is 90 seconds Sep 28 12:00:12.744346 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: mapped 403077 bytes (393 KB) for 2 cores Sep 28 12:00:12.744346 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: *** Operational MODE: preforking *** Sep 28 12:00:12.744625 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: *** uWSGI is running in multiple interpreter mode *** Sep 28 12:00:12.744625 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: spawned uWSGI master process (pid: 81580) Sep 28 12:00:12.744829 dv-cinder-859193-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 28 12:00:12.749338 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: spawned uWSGI worker 1 (pid: 81592, cores: 1) Sep 28 12:00:12.749338 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: spawned uWSGI worker 2 (pid: 81593, cores: 1) Sep 28 12:00:12.749338 dv-cinder-859193-2-iscsi devstack@keystone.service[81580]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 28 12:00:17.003669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 28 12:00:17.003669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: from cryptography.utils import int_from_bytes Sep 28 12:00:17.004506 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 28 12:00:17.004506 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: from cryptography.utils import int_from_bytes Sep 28 12:00:17.006639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 28 12:00:17.006639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: from cryptography.utils import int_from_bytes Sep 28 12:00:17.007094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 28 12:00:17.007094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: from cryptography.utils import int_from_bytes Sep 28 12:00:17.054080 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 28 12:00:17.054272 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 28 12:00:17.054474 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] command line args: [] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 28 12:00:17.055339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 28 12:00:17.055527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ================================================================================ {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 28 12:00:17.055788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] admin_token = **** {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.056037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] config_dir = [] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.056229 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.056437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] config_source = [] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.056621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] debug = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.056808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 28 12:00:17.056892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.057070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 28 12:00:17.057193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] command line args: [] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 28 12:00:17.057267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.057367 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.057442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 28 12:00:17.057612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ================================================================================ {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 28 12:00:17.057694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.057860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.057942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] admin_token = **** {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.058093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] list_limit = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.058169 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] config_dir = [] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.058272 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] log_config_append = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.058350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.058453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.058524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] config_source = [] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.058621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] debug = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.058696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] log_dir = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.058802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] log_file = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.058877 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.059032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.059123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.059225 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.059372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.059457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.059539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.059725 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.059827 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.059905 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.059989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] list_limit = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.060069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.060142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] log_config_append = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.060264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.060388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.060473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] log_dir = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.060552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.060630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] log_file = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.060866 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.060970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.061043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.061127 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.061251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.061344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.061487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.061587 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.061709 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.061808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.061881 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.061956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.062119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.062247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.062365 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.062601 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.062702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.26/identity {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.062831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.062925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] publish_errors = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.063059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.063159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.063291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.063392 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.063486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.063572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.063736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.063823 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.063971 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.064062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.064209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] standard_threads = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.064477 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.064572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.26/identity {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.064705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] publish_errors = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.064807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.064883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.064942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.065055 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] use_journal = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.065161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.065274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] use_json = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.065359 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.065432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] use_stderr = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.065548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] use_syslog = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.065641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.065779 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.065873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.065996 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] standard_threads = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.066070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.066241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.066352 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.066466 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.066575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.066696 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.066775 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.066846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] use_journal = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.066954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] use_json = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.067030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.067109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] use_stderr = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.067191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.067297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] use_syslog = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.067434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.067512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 28 12:00:17.067673 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.067757 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.067923 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.068041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.068172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.068251 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.068432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.068522 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.068670 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.068750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.068916 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.069002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.069152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.069235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.069384 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.069498 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.069631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.069710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.069867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.069945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.070108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.070186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.070363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.070448 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.070609 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.070688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.070844 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.070930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.071090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.071168 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.071337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.071416 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.071564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.071639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.071813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.071898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.072046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.072134 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.072279 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.072376 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.072535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.072613 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.072772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.072865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.073023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.073107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.073272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.073349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.073501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.073625 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.073735 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.073814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.073961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.074039 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.074190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.074275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.074424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.074502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.074668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.074746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.074905 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.074989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.075131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.075208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.075359 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.075437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.075589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.075677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.075819 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.075897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.076052 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.076129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.076283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.076395 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.076541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.076623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.076774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.076858 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.077006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.077160 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.077236 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.077304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.077446 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.077524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.077675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.077761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.077904 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.077982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.078138 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.078217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.078365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.078451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.078601 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.078678 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.078836 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.078918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.079062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.079147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.079295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.079365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.079529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.079596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.079757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.079832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.079993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.080065 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.080229 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.080440 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.080595 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.080704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.080845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.080962 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.080962 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.081190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.081303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.081422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.081560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.081668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.081781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.081925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.082044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.082184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.082305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.082443 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.082551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.082668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.082807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.082915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.083030 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.083170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.083278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.083385 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.083522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.083630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.083746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.083897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.084014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.084152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.084263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.084381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.084535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.084647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.084759 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.084871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.084955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.085027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.085145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.085221 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.085293 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.085369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.085441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.085550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.085630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.085703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.085773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.085843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.085913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.086006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.086142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.086292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.086375 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.086503 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.086613 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.086755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.086841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.086996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.087075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.087500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.087615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.087747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.087839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.087989 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.088074 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.088229 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.088332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.088492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.088649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.088725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.088796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.088947 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.089027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.089180 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.089268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.089417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.089498 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.089649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.089726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.089886 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.089964 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.090116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.090228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.090346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.090424 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.090574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.090659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.090808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.090889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.091038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.091117 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.091275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.091354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.091511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.091603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] auth.external = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.091749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.091827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.091984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.092088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.092239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.092367 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.092513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] auth.password = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.092593 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.092746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] auth.token = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.092827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.092976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] auth.external = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.093053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.093206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.093286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.093452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.093530 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.093686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.093765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.093933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] auth.password = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.094018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.094163 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] auth.token = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.094242 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.094396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.094473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.094628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.094709 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] credential.caching = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.094873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.094951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.095101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.095179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.095367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.095483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.095589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.095666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.095816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.095894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.096051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] credential.caching = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.096132 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.096296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.096424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.096569 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.096658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.096805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.096885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.097043 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.097122 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.097285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.097364 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.097519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.097599 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.097763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.097842 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.098014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.098093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.098249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.098331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.098500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.098579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] federation.caching = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.098738 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.098824 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.098987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.099069 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.099235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.099314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.099465 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.099544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.099716 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.099797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.099950 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] federation.caching = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.100027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.100186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.100274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.100452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.100531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.100700 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.100777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.100930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.101020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.101177 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.101255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.101410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.101488 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity.caching = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.102173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.102261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.102409 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.102487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.102651 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.102729 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.102885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.103038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.103121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.103194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.103351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.103429 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity.caching = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.103588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.103681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.103831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.103914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.104064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.104145 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.104297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.104413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.104552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.104630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.104786 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.104894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.105021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.105113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.105267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.105345 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.105514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.105590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.105743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.105829 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.105974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.106073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.106206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.106290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.106445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.106546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.106689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.106766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.106922 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.106998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.107162 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.107249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.107395 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.107472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.107630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.107708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.107868 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.107957 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.108100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.108176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.108346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.108427 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.108587 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.108680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.108816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.108895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.109053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.109130 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.109289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.109374 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.109523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.109599 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.109767 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.109844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.109996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.110074 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.110235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.110313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.110466 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.110543 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.110695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.110773 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.110931 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.111008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.111171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.111247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.111398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.111477 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.111635 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.111743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.111863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.111999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.112084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.112251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.112358 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.112512 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.112590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.112746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.112835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.112998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.113118 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.113250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.113370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.113489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.113619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.113728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.113853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.113966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.114084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.114200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.114333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.114440 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.114580 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.114700 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.114816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.114953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.115040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.115187 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.115264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.115436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.115513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.115683 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.115768 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.115919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.115996 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.116150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.116226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.116409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.116503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.116648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.116727 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.116887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.116963 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.117118 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.117267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.117901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.117983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.118139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.118218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.118373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.118460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.118610 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.118687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.118840 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.118916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.119082 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.119167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.119326 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.119402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.119563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.119639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.119803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.119893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.120044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.120122 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.120271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.120361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.120539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.120635 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.120784 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.120862 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.121021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.121116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.121256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.121346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.121491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.121568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.121732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.121808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.121969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.122054 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.122204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.122280 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.122439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.122515 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.122673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.122756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.122916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.122993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.123147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.123226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.123379 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.123462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.123615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.123692 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] resource.caching = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.123859 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.123936 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.124100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.124186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.124348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.124429 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.124587 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.124664 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.124828 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] resource.caching = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.124917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.125069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.125146 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.125315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.125401 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.125548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.125633 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.125789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.125867 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.126026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] role.caching = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.126103 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.126250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.126323 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] role.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.126489 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.126566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.126725 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.126802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.126963 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.127041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.127198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] role.caching = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.127274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.127444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.127519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] role.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.127670 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.127747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.127917 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.127994 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.128163 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.128247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.128432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.128514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.128668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.128745 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.128913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.128989 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.129159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.129237 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.129401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.129477 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.129644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.129720 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.129875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.130037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.130115 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.130277 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.130354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.130528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.130614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.130778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.130855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.131029 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.131194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.131271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.131430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.131508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.131678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.131755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.131927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.132011 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.132174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.132251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.132443 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.132526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.132677 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.132777 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.132936 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.133014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.133191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.133268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.133428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.133513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.133661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.133739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.133892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.133969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.134124 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.134209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.134353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.134430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.134591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] token.caching = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.134668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.134828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.134913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.135070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.135147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.135304 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.135380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.135534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.135619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.135770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] token.caching = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.135847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.136005 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.136082 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.136247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.136361 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.136506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.136583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.136753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.136829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.136991 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.137076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.137225 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.137307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.137459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.137536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.137699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.137783 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.137945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.138022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.138180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.138257 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.138417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.138502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.138650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.138729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.138895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.138971 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.connection = **** {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.139157 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.139233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.139383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.139469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.139623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.139702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.139864 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.139940 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.140100 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.140178 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.connection = **** {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.140362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.140444 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.140602 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.140680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.140839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.140921 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.141075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.141153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.141309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.141386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.141544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.141621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.141779 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.141857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.142011 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.142086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.142253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.142328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.142485 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.142562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.142719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.142795 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.142950 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.143026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.143184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.143261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.143355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81593) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 28 12:00:17.143432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.143647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.143875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.144113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.144364 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 28 12:00:17.144537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81592) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 28 12:00:17.173233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.173462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.173688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.175019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.175248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.175472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.182657 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.184407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.210569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.210803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.216017 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.216249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.264645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.265270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.313674 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.313907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.314033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.314266 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.469784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.470016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.470234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.470320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.470454 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.470561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.470672 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.470755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.470888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.470970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.471102 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.471433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.495445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.495678 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.495857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.496013 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.496150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.496283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.496444 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.496780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.496929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.497053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.497181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.497311 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.497439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.497439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.508356 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.508599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.521370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.521553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.521776 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.521995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.522211 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.522426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.522641 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81593) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.525574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.525809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.526033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.526253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.526470 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.526684 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.526899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81592) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 28 12:00:17.540541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.540752 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 28 12:00:17.541095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.541177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.541296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 28 12:00:17.541394 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 28 12:00:17.541569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.541679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.541830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.541913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 28 12:00:17.542090 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.542173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.542348 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.542429 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.542632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.542891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.552062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.552374 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.552462 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.552650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.552733 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.552926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.553010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.553207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.559894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.560114 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.562812 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.563015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.563099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.563296 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.563379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.563578 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.563673 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.563859 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.563943 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.564141 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.564224 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.564428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.564515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.564700 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.564783 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.564973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.565065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.565241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.565321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.565509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.565589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.565772 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.588927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.589336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.590388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.590671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.590803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.591090 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.595276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.595561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.595733 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.595828 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.596015 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.596297 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.602691 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.602970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.603193 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.603481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.608612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.608908 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.609177 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.609295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.609469 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.609563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.609745 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.609825 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.610019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.610098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.610291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.610372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.610560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.610640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.610831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.611102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.625900 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.626163 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.626586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.626864 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.630001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.630741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.633253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.633550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.634010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.634310 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.637722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.638503 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.640865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.641686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.644053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.644371 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.644658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.644915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.644996 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.645219 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.645312 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.645509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.645591 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.645796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.646074 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.646371 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.661609 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.661893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.662162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.662426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.662542 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.662834 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.663107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.663375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.669770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.670762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.671312 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.672331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.672852 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.673127 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.673855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.674134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.675750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.676032 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.676325 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.676626 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.676906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.677025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.677192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.677477 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.677765 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.678178 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.688056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.688357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.688640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.688921 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.689239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.689333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.689526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.689607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.689812 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.689893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.690105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.690486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.690775 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.691058 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.705129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.706325 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.708001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.709242 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.711519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.712932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.713295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.714521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.714752 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.715034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.715435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.716025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.716334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.716741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.721747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.723080 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.723189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.724566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.726071 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.726456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.726795 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.727076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.727535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.727934 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.728231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.728538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.734809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.735098 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.735380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.736259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.736575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.736862 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.742732 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81592) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.743026 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.743308 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.743584 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.743857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.744126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.744206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81593) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 28 12:00:17.744422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.744508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.744702 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.744783 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.744981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.745073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.745249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.745340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.745532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.745622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.745793 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.745874 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.746075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.746195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.746346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.746473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81592) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.746630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.746914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.747180 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.747464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.747732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81593) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 28 12:00:17.781027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55ce0b7f09c0 pid: 81592 (default app) Sep 28 12:00:17.782329 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55ce0b7f09c0 pid: 81593 (default app) Sep 28 12:00:17.784541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e815e3ba-0c0e-46b4-8f90-58705891d516 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:17.784830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e815e3ba-0c0e-46b4-8f90-58705891d516 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:17.785102 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e815e3ba-0c0e-46b4-8f90-58705891d516 None None] PATH_INFO: `/v3` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:17.790177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1/1] 10.222.0.26 () {50 vars in 791 bytes} [Wed Sep 28 12:00:17 2022] GET /identity/v3/ => generated 255 bytes in 9 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:24.523248 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c665d536-687e-43b2-9d12-476dce0e5a18 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:24.523653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c665d536-687e-43b2-9d12-476dce0e5a18 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:24.523802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c665d536-687e-43b2-9d12-476dce0e5a18 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:24.528049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1/2] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:24 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:24.533654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e264bd-4437-47da-9f61-99ca198f9411 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:24.534109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e264bd-4437-47da-9f61-99ca198f9411 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:24.534247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e264bd-4437-47da-9f61-99ca198f9411 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:24.604478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: INFO dbcounter [None req-44e264bd-4437-47da-9f61-99ca198f9411 None None] Registered counter for database keystone Sep 28 12:00:24.623533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG oslo_db.sqlalchemy.engines [None req-44e264bd-4437-47da-9f61-99ca198f9411 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81592) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 28 12:00:24.629676 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writer thread running {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 28 12:00:24.806736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG passlib.handlers.bcrypt [None req-44e264bd-4437-47da-9f61-99ca198f9411 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81592) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 28 12:00:24.807123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG passlib.handlers.bcrypt [None req-44e264bd-4437-47da-9f61-99ca198f9411 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81592) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 28 12:00:24.864697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-44e264bd-4437-47da-9f61-99ca198f9411 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:24.866303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-44e264bd-4437-47da-9f61-99ca198f9411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:24.991249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 2/3] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:24 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 459 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 28 12:00:25.430853 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c665d536-687e-43b2-9d12-476dce0e5a18 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:25.500269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: INFO dbcounter [None req-c665d536-687e-43b2-9d12-476dce0e5a18 None None] Registered counter for database keystone Sep 28 12:00:25.521701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG oslo_db.sqlalchemy.engines [None req-c665d536-687e-43b2-9d12-476dce0e5a18 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81593) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 28 12:00:25.528505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writer thread running {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 28 12:00:25.645560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c665d536-687e-43b2-9d12-476dce0e5a18 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:25.649821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56320fe5-5fb7-4173-82fe-d84fb659d662 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:25.650446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56320fe5-5fb7-4173-82fe-d84fb659d662 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:25.650731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56320fe5-5fb7-4173-82fe-d84fb659d662 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:25.651006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56320fe5-5fb7-4173-82fe-d84fb659d662 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:25.659104 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56320fe5-5fb7-4173-82fe-d84fb659d662 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:25.710365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: WARNING keystone.common.rbac_enforcer.enforcer [None req-56320fe5-5fb7-4173-82fe-d84fb659d662 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 28 12:00:25.710697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56320fe5-5fb7-4173-82fe-d84fb659d662 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:25.720067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 2/4] 10.222.0.26 () {58 vars in 1324 bytes} [Wed Sep 28 12:00:25 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 28 12:00:25.726018 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e264bd-4437-47da-9f61-99ca198f9411 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:25.743048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e264bd-4437-47da-9f61-99ca198f9411 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:25.747714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ef42ef-1dee-4ad6-a0af-d66ccd0f50cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:25.748836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ef42ef-1dee-4ad6-a0af-d66ccd0f50cb admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:25.749116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ef42ef-1dee-4ad6-a0af-d66ccd0f50cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:25.749385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ef42ef-1dee-4ad6-a0af-d66ccd0f50cb admin admin] PATH_INFO: `/v3/projects/8d6b5c1f82154555b8d48a213cdda1ed` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:25.750955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ef42ef-1dee-4ad6-a0af-d66ccd0f50cb admin admin] RBAC: Authorizing `identity:get_project(project_id=8d6b5c1f82154555b8d48a213cdda1ed)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:25.803446 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: WARNING keystone.common.rbac_enforcer.enforcer [None req-89ef42ef-1dee-4ad6-a0af-d66ccd0f50cb admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 28 12:00:25.803791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ef42ef-1dee-4ad6-a0af-d66ccd0f50cb admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:25.808795 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 3/5] 10.222.0.26 () {56 vars in 1207 bytes} [Wed Sep 28 12:00:25 2022] GET /identity/v3/projects/8d6b5c1f82154555b8d48a213cdda1ed => generated 344 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:27.933643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf2abea-1680-46ff-99f8-a7cbd62793d2 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:27.934165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf2abea-1680-46ff-99f8-a7cbd62793d2 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:27.934165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf2abea-1680-46ff-99f8-a7cbd62793d2 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:27.938246 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 3/6] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:27 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:00:27.943590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d701238d-dc1e-4597-8ee1-c6f5537d92d6 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:27.943957 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d701238d-dc1e-4597-8ee1-c6f5537d92d6 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:27.944151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d701238d-dc1e-4597-8ee1-c6f5537d92d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:27.965074 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-d701238d-dc1e-4597-8ee1-c6f5537d92d6 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:27.966586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d701238d-dc1e-4597-8ee1-c6f5537d92d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:27.974388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 4/7] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:27 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 28 12:00:28.255133 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf2abea-1680-46ff-99f8-a7cbd62793d2 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:28.267494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf2abea-1680-46ff-99f8-a7cbd62793d2 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:28.271615 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85d8c3e-ab79-4ec5-a129-03d9934d5b46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:28.272538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85d8c3e-ab79-4ec5-a129-03d9934d5b46 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:28.272818 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85d8c3e-ab79-4ec5-a129-03d9934d5b46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:28.273653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85d8c3e-ab79-4ec5-a129-03d9934d5b46 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:28.281102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85d8c3e-ab79-4ec5-a129-03d9934d5b46 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:28.282844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85d8c3e-ab79-4ec5-a129-03d9934d5b46 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:28.291396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 4/8] 10.222.0.26 () {58 vars in 1324 bytes} [Wed Sep 28 12:00:28 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 28 12:00:28.296679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d701238d-dc1e-4597-8ee1-c6f5537d92d6 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:28.305739 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d701238d-dc1e-4597-8ee1-c6f5537d92d6 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:28.310095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a75b23-ceaa-4dbd-bafb-bcd290c9deeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:28.310718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a75b23-ceaa-4dbd-bafb-bcd290c9deeb admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:28.311003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a75b23-ceaa-4dbd-bafb-bcd290c9deeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:28.311281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a75b23-ceaa-4dbd-bafb-bcd290c9deeb admin admin] PATH_INFO: `/v3/users/f82a94e2366b45ff82ac58061224a3a2` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:28.312642 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a75b23-ceaa-4dbd-bafb-bcd290c9deeb admin admin] RBAC: Authorizing `identity:get_user(user_id=f82a94e2366b45ff82ac58061224a3a2)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:28.314461 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a75b23-ceaa-4dbd-bafb-bcd290c9deeb admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:28.315969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 5/9] 10.222.0.26 () {56 vars in 1198 bytes} [Wed Sep 28 12:00:28 2022] GET /identity/v3/users/f82a94e2366b45ff82ac58061224a3a2 => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:30.525411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dd1d4e-d740-4045-ae32-7bd4d5b55310 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:30.525928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dd1d4e-d740-4045-ae32-7bd4d5b55310 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:30.526017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dd1d4e-d740-4045-ae32-7bd4d5b55310 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:30.527272 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 5/10] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:30.533414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e482263-f193-4995-82c9-27eaf410a4cd None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:30.533916 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e482263-f193-4995-82c9-27eaf410a4cd None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:30.534186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e482263-f193-4995-82c9-27eaf410a4cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:30.547129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d981f-46bf-434a-a7fe-ce081ce47609 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:30.547415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d981f-46bf-434a-a7fe-ce081ce47609 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:30.547693 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d981f-46bf-434a-a7fe-ce081ce47609 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:30.549045 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 6/11] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:00:30.553210 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-9e482263-f193-4995-82c9-27eaf410a4cd None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:30.554074 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f6b6a2-8cdb-43b3-881e-433506d07b2f None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:30.554928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f6b6a2-8cdb-43b3-881e-433506d07b2f None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:30.555028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9e482263-f193-4995-82c9-27eaf410a4cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:30.555209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f6b6a2-8cdb-43b3-881e-433506d07b2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:30.596573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG passlib.handlers.bcrypt [None req-72f6b6a2-8cdb-43b3-881e-433506d07b2f None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81593) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 28 12:00:30.596958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG passlib.handlers.bcrypt [None req-72f6b6a2-8cdb-43b3-881e-433506d07b2f None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81593) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 28 12:00:30.636741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 6/12] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:00:30.646131 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-72f6b6a2-8cdb-43b3-881e-433506d07b2f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:30.647732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-72f6b6a2-8cdb-43b3-881e-433506d07b2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:30.745140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 7/13] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:00:30.903648 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e482263-f193-4995-82c9-27eaf410a4cd None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:30.927115 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:31.008150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e482263-f193-4995-82c9-27eaf410a4cd None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:31.008382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f6b6a2-8cdb-43b3-881e-433506d07b2f None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:31.040777 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53bf970-5ff8-4ae5-a2d2-f6196706655e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:31.041408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53bf970-5ff8-4ae5-a2d2-f6196706655e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:31.041687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53bf970-5ff8-4ae5-a2d2-f6196706655e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:31.041961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53bf970-5ff8-4ae5-a2d2-f6196706655e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:31.052751 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53bf970-5ff8-4ae5-a2d2-f6196706655e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-a53bf970-5ff8-4ae5-a2d2-f6196706655e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:31.062171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:31.063051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:31.063845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:31.063845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:31.063845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:31.063845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:31.063845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:31.063845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:31.063845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:31.063845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:31.063845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:31.063845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:31.063845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:00:31.063845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:00:31.063845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 7/14] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:00:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:31.069085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53bf970-5ff8-4ae5-a2d2-f6196706655e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:31.109538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f6b6a2-8cdb-43b3-881e-433506d07b2f None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:31.111772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:31.147060 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b36b3d6-c86a-46fc-a408-582e654f9777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:31.147958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b36b3d6-c86a-46fc-a408-582e654f9777 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:31.148234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b36b3d6-c86a-46fc-a408-582e654f9777 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:31.148540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b36b3d6-c86a-46fc-a408-582e654f9777 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:31.149543 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:31.169428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53bf970-5ff8-4ae5-a2d2-f6196706655e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:31.203752 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5791add9-4d69-42fc-b8d9-d3286864160c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:31.204403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5791add9-4d69-42fc-b8d9-d3286864160c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:31.204684 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5791add9-4d69-42fc-b8d9-d3286864160c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:31.204985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5791add9-4d69-42fc-b8d9-d3286864160c None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:31.205656 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5791add9-4d69-42fc-b8d9-d3286864160c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:31.205964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5791add9-4d69-42fc-b8d9-d3286864160c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:31.207363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5791add9-4d69-42fc-b8d9-d3286864160c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:31.215125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b36b3d6-c86a-46fc-a408-582e654f9777 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:31.216650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b36b3d6-c86a-46fc-a408-582e654f9777 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:31.219048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 8/15] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:00:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:00:31.224829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5791add9-4d69-42fc-b8d9-d3286864160c None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:31.319389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 8/16] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:00:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 28 12:00:31.325080 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b36b3d6-c86a-46fc-a408-582e654f9777 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:31.326203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5791add9-4d69-42fc-b8d9-d3286864160c None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:31.327685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:31.359881 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b661b51-9a0a-44d6-8dcb-5781019cd8de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:31.360544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b661b51-9a0a-44d6-8dcb-5781019cd8de None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:31.360823 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b661b51-9a0a-44d6-8dcb-5781019cd8de None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:31.361092 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b661b51-9a0a-44d6-8dcb-5781019cd8de None admin] PATH_INFO: `/v3/users/f82a94e2366b45ff82ac58061224a3a2` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:31.362249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b661b51-9a0a-44d6-8dcb-5781019cd8de None admin] RBAC: Authorizing `identity:get_user(user_id=f82a94e2366b45ff82ac58061224a3a2)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:31.363703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b661b51-9a0a-44d6-8dcb-5781019cd8de None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:31.365292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 9/17] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:00:31 2022] GET /identity/v3/users/f82a94e2366b45ff82ac58061224a3a2 => generated 247 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:00:31.371184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b661b51-9a0a-44d6-8dcb-5781019cd8de None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:31.384927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:31.422565 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b36b3d6-c86a-46fc-a408-582e654f9777 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:31.456695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c8347d-e23c-4728-8b59-2ee3007e76ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:31.457543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c8347d-e23c-4728-8b59-2ee3007e76ea None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:31.457821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c8347d-e23c-4728-8b59-2ee3007e76ea None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:31.458098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c8347d-e23c-4728-8b59-2ee3007e76ea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:31.466307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b661b51-9a0a-44d6-8dcb-5781019cd8de None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:31.476959 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c8347d-e23c-4728-8b59-2ee3007e76ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:31.478417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c8347d-e23c-4728-8b59-2ee3007e76ea None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-23c8347d-e23c-4728-8b59-2ee3007e76ea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:00:31.500156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:31.501228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:00:31.502060 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:00:31.502060 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:00:31.502060 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:00:31.502060 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 9/18] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:00:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:00:31.503655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee36892-5bf3-417c-86d8-fc56637ee45c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:31.504288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee36892-5bf3-417c-86d8-fc56637ee45c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:31.504597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee36892-5bf3-417c-86d8-fc56637ee45c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:31.504871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee36892-5bf3-417c-86d8-fc56637ee45c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:31.505992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee36892-5bf3-417c-86d8-fc56637ee45c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:31.506961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c8347d-e23c-4728-8b59-2ee3007e76ea None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:31.507468 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee36892-5bf3-417c-86d8-fc56637ee45c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:31.509026 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 10/19] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:00:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:00:31.514486 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee36892-5bf3-417c-86d8-fc56637ee45c None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:31.565502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:31.603865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:31.610165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c8347d-e23c-4728-8b59-2ee3007e76ea None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:31.623269 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee36892-5bf3-417c-86d8-fc56637ee45c None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:31.642971 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d03bb9-df79-438e-8a1f-4c3331ce2bcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:31.643575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d03bb9-df79-438e-8a1f-4c3331ce2bcd None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:31.643882 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d03bb9-df79-438e-8a1f-4c3331ce2bcd None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:31.644151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d03bb9-df79-438e-8a1f-4c3331ce2bcd None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:31.644999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d03bb9-df79-438e-8a1f-4c3331ce2bcd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:31.645621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d03bb9-df79-438e-8a1f-4c3331ce2bcd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:31.646988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d03bb9-df79-438e-8a1f-4c3331ce2bcd None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:31.656149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacaab46-33bf-4198-9763-eb2551843df3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:31.656781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacaab46-33bf-4198-9763-eb2551843df3 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:31.657060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacaab46-33bf-4198-9763-eb2551843df3 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:31.657327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacaab46-33bf-4198-9763-eb2551843df3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:31.658044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacaab46-33bf-4198-9763-eb2551843df3 None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.domain.id=default, user.id=f82a94e2366b45ff82ac58061224a3a2) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:31.658347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacaab46-33bf-4198-9763-eb2551843df3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:31.659756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacaab46-33bf-4198-9763-eb2551843df3 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:31.664605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 10/20] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:00:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:00:31.669883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 11/21] 10.222.0.26 () {56 vars in 1322 bytes} [Wed Sep 28 12:00:31 2022] GET /identity/v3/role_assignments?user.id=f82a94e2366b45ff82ac58061224a3a2&scope.domain.id=default&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 233 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:00:33.977046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2492524e-3d63-4e1c-8144-62ed0d47fb50 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:33.977553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2492524e-3d63-4e1c-8144-62ed0d47fb50 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:33.977625 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2492524e-3d63-4e1c-8144-62ed0d47fb50 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:33.978902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 11/22] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:33.984292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586b28a3-d68b-4a41-85eb-b4c841947b73 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:33.984720 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586b28a3-d68b-4a41-85eb-b4c841947b73 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:33.984894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586b28a3-d68b-4a41-85eb-b4c841947b73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:34.009276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-586b28a3-d68b-4a41-85eb-b4c841947b73 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:34.010478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-586b28a3-d68b-4a41-85eb-b4c841947b73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:34.105771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 12/23] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:34.384225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2d90ac-b2f3-44c1-b0d3-bf75d8738c7b None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:34.384532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2d90ac-b2f3-44c1-b0d3-bf75d8738c7b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:34.384796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2d90ac-b2f3-44c1-b0d3-bf75d8738c7b None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:34.386310 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 12/24] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:34.391537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389a70f4-f59a-4bfc-9399-69c45154d932 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:34.391838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389a70f4-f59a-4bfc-9399-69c45154d932 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:34.392124 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389a70f4-f59a-4bfc-9399-69c45154d932 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:34.417436 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-389a70f4-f59a-4bfc-9399-69c45154d932 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:34.419178 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-389a70f4-f59a-4bfc-9399-69c45154d932 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:34.514830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 13/25] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:34.605618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2d90ac-b2f3-44c1-b0d3-bf75d8738c7b None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:34.631878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acbb676-fb7d-4e90-ae4a-8c929399e6e3 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:34.633038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acbb676-fb7d-4e90-ae4a-8c929399e6e3 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:34.633491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acbb676-fb7d-4e90-ae4a-8c929399e6e3 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:34.635178 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 14/26] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:00:34.641468 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4227f2-6ac7-46f9-baf5-99c3e9896337 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:34.641974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4227f2-6ac7-46f9-baf5-99c3e9896337 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:34.642426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4227f2-6ac7-46f9-baf5-99c3e9896337 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:34.653422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:34.667341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-bf4227f2-6ac7-46f9-baf5-99c3e9896337 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:34.668905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-bf4227f2-6ac7-46f9-baf5-99c3e9896337 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:34.783004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 15/27] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:00:34.810004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2d90ac-b2f3-44c1-b0d3-bf75d8738c7b None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:34.843850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:34.871606 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4227f2-6ac7-46f9-baf5-99c3e9896337 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:34.872743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1105ecbd-31d1-4cd1-9737-580c250f08bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:34.873550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105ecbd-31d1-4cd1-9737-580c250f08bc None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:34.874045 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105ecbd-31d1-4cd1-9737-580c250f08bc None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:34.874584 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105ecbd-31d1-4cd1-9737-580c250f08bc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:34.894631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1105ecbd-31d1-4cd1-9737-580c250f08bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-1105ecbd-31d1-4cd1-9737-580c250f08bc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:34.905149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:34.905982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:34.906730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:34.906730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:34.906730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:34.906730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:34.906730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:34.906730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:34.906730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:34.906730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:34.906730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:34.906730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:34.906730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:00:34.906730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:00:34.907286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 13/28] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:00:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:34.913508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1105ecbd-31d1-4cd1-9737-580c250f08bc None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:35.061529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1105ecbd-31d1-4cd1-9737-580c250f08bc None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:35.099324 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4227f2-6ac7-46f9-baf5-99c3e9896337 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:35.116237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c4ea59-25c6-4abc-8cd2-721a7ec6ffc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:35.117694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c4ea59-25c6-4abc-8cd2-721a7ec6ffc9 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:35.118137 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c4ea59-25c6-4abc-8cd2-721a7ec6ffc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:35.118563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c4ea59-25c6-4abc-8cd2-721a7ec6ffc9 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:35.119395 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c4ea59-25c6-4abc-8cd2-721a7ec6ffc9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:35.119840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c4ea59-25c6-4abc-8cd2-721a7ec6ffc9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:35.122789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c4ea59-25c6-4abc-8cd2-721a7ec6ffc9 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:35.146164 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 14/29] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:00:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:35.148778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21640b11-da99-4434-8f81-83de24ef9cd4 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:35.150533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21640b11-da99-4434-8f81-83de24ef9cd4 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:35.150993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21640b11-da99-4434-8f81-83de24ef9cd4 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:35.152538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 15/30] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:35.154419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21640b11-da99-4434-8f81-83de24ef9cd4 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:35.171844 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303cc733-50a8-4c1f-8ee9-daac5859e72d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:35.172758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303cc733-50a8-4c1f-8ee9-daac5859e72d None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:35.173194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303cc733-50a8-4c1f-8ee9-daac5859e72d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:35.173609 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303cc733-50a8-4c1f-8ee9-daac5859e72d None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:35.174441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303cc733-50a8-4c1f-8ee9-daac5859e72d None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:35.175987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303cc733-50a8-4c1f-8ee9-daac5859e72d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:35.185716 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:35.229748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:35.295131 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21640b11-da99-4434-8f81-83de24ef9cd4 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:35.328368 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 16/31] 10.222.0.26 () {60 vars in 1131 bytes} [Wed Sep 28 12:00:34 2022] POST /identity/v3/roles => generated 219 bytes in 457 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:00:35.330321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5802b6e-bd5d-4fcc-a19e-787ea9b77d60 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:35.330818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5802b6e-bd5d-4fcc-a19e-787ea9b77d60 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:35.331300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5802b6e-bd5d-4fcc-a19e-787ea9b77d60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:35.341281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030abd9e-ab11-45af-91a3-4bbb41bf9e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:35.342338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030abd9e-ab11-45af-91a3-4bbb41bf9e39 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:35.342842 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030abd9e-ab11-45af-91a3-4bbb41bf9e39 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:35.344655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030abd9e-ab11-45af-91a3-4bbb41bf9e39 None admin] PATH_INFO: `/v3/users/f82a94e2366b45ff82ac58061224a3a2` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:35.348579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030abd9e-ab11-45af-91a3-4bbb41bf9e39 None admin] RBAC: Authorizing `identity:get_user(user_id=f82a94e2366b45ff82ac58061224a3a2)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:35.357589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030abd9e-ab11-45af-91a3-4bbb41bf9e39 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:35.360258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 16/32] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:00:35 2022] GET /identity/v3/users/f82a94e2366b45ff82ac58061224a3a2 => generated 247 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:35.361684 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030abd9e-ab11-45af-91a3-4bbb41bf9e39 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:35.372058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-b5802b6e-bd5d-4fcc-a19e-787ea9b77d60 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:35.379375 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b5802b6e-bd5d-4fcc-a19e-787ea9b77d60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:35.489811 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030abd9e-ab11-45af-91a3-4bbb41bf9e39 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:35.500748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:35.528152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15db43ba-d6e6-4c8f-9a7c-65f209474bdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:35.529059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15db43ba-d6e6-4c8f-9a7c-65f209474bdd None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:35.529509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15db43ba-d6e6-4c8f-9a7c-65f209474bdd None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:35.529927 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15db43ba-d6e6-4c8f-9a7c-65f209474bdd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:35.542176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15db43ba-d6e6-4c8f-9a7c-65f209474bdd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:35.543897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15db43ba-d6e6-4c8f-9a7c-65f209474bdd None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-15db43ba-d6e6-4c8f-9a7c-65f209474bdd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:00:35.559521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:35.560432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:00:35.561269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:00:35.561269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:00:35.561269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:00:35.561269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 17/33] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:00:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:35.561473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15db43ba-d6e6-4c8f-9a7c-65f209474bdd None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:35.563659 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 17/34] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:35.575620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5802b6e-bd5d-4fcc-a19e-787ea9b77d60 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:35.674395 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15db43ba-d6e6-4c8f-9a7c-65f209474bdd None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:35.713318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d84f54-82e6-474d-8738-845f4f7852b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:35.714207 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d84f54-82e6-474d-8738-845f4f7852b8 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:35.714656 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d84f54-82e6-474d-8738-845f4f7852b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:35.715118 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d84f54-82e6-474d-8738-845f4f7852b8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:35.716421 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d84f54-82e6-474d-8738-845f4f7852b8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:35.718102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d84f54-82e6-474d-8738-845f4f7852b8 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:35.720189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 18/35] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:00:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:35.726629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d84f54-82e6-474d-8738-845f4f7852b8 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:35.756436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:35.786259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5802b6e-bd5d-4fcc-a19e-787ea9b77d60 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:35.832321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:35.842902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d84f54-82e6-474d-8738-845f4f7852b8 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:35.861036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a13d177-e157-401b-ab1e-3cb6d5370c2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:35.861822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a13d177-e157-401b-ab1e-3cb6d5370c2d None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:35.862215 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a13d177-e157-401b-ab1e-3cb6d5370c2d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:35.862602 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a13d177-e157-401b-ab1e-3cb6d5370c2d None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:35.863376 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a13d177-e157-401b-ab1e-3cb6d5370c2d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:35.863787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a13d177-e157-401b-ab1e-3cb6d5370c2d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:35.865467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a13d177-e157-401b-ab1e-3cb6d5370c2d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:35.880658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63741a4a-8f00-4093-9cf0-cfb78e2c3928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:35.881546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63741a4a-8f00-4093-9cf0-cfb78e2c3928 None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:35.882495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63741a4a-8f00-4093-9cf0-cfb78e2c3928 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:35.882907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63741a4a-8f00-4093-9cf0-cfb78e2c3928 None admin] PATH_INFO: `/v3/domains/default/users/f82a94e2366b45ff82ac58061224a3a2/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:35.885152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63741a4a-8f00-4093-9cf0-cfb78e2c3928 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=f82a94e2366b45ff82ac58061224a3a2, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:35.886857 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63741a4a-8f00-4093-9cf0-cfb78e2c3928 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:35.896532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 18/36] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:00:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:35.898569 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a13d177-e157-401b-ab1e-3cb6d5370c2d None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:35.996391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a13d177-e157-401b-ab1e-3cb6d5370c2d None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:36.018728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 19/37] 10.222.0.26 () {58 vars in 1361 bytes} [Wed Sep 28 12:00:35 2022] PUT /identity/v3/domains/default/users/f82a94e2366b45ff82ac58061224a3a2/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:00:36.024494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94abf4c7-146d-4d42-97e7-2503ede59e05 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:36.024494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94abf4c7-146d-4d42-97e7-2503ede59e05 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:36.024494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94abf4c7-146d-4d42-97e7-2503ede59e05 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:36.026554 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 20/38] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:36 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:36.027276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94abf4c7-146d-4d42-97e7-2503ede59e05 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:36.042637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca94064-307b-4f8d-b7c8-96629d3019a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:36.043257 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca94064-307b-4f8d-b7c8-96629d3019a3 None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:36.043543 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca94064-307b-4f8d-b7c8-96629d3019a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:36.043822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca94064-307b-4f8d-b7c8-96629d3019a3 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:36.044540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca94064-307b-4f8d-b7c8-96629d3019a3 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:36.046227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca94064-307b-4f8d-b7c8-96629d3019a3 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:36.063749 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 19/39] 10.222.0.26 () {60 vars in 1131 bytes} [Wed Sep 28 12:00:35 2022] POST /identity/v3/roles => generated 225 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:00:36.065232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36f3497-5aa7-45e6-8e65-956250b0799c None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:36.065523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36f3497-5aa7-45e6-8e65-956250b0799c None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:36.065802 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36f3497-5aa7-45e6-8e65-956250b0799c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:36.088504 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-a36f3497-5aa7-45e6-8e65-956250b0799c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:36.089809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a36f3497-5aa7-45e6-8e65-956250b0799c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:36.184129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:36.209231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:36.233274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94abf4c7-146d-4d42-97e7-2503ede59e05 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:36.239478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 20/40] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:36.240873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25decabf-ea12-4ab7-afe3-b73ab35de3a7 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:36.241166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25decabf-ea12-4ab7-afe3-b73ab35de3a7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:36.241980 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25decabf-ea12-4ab7-afe3-b73ab35de3a7 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:36.243438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 21/41] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:36.254156 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4072ff-7f00-4bce-ae9b-b4769a98eb70 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:36.254590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4072ff-7f00-4bce-ae9b-b4769a98eb70 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:36.255120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4072ff-7f00-4bce-ae9b-b4769a98eb70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:36.281241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-ba4072ff-7f00-4bce-ae9b-b4769a98eb70 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:36.282469 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-ba4072ff-7f00-4bce-ae9b-b4769a98eb70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:36.284031 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11933c9d-e096-4d3e-9bde-2f202c531ad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:36.284735 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11933c9d-e096-4d3e-9bde-2f202c531ad1 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:36.285025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11933c9d-e096-4d3e-9bde-2f202c531ad1 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:36.285295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11933c9d-e096-4d3e-9bde-2f202c531ad1 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:36.285959 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11933c9d-e096-4d3e-9bde-2f202c531ad1 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:36.287604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11933c9d-e096-4d3e-9bde-2f202c531ad1 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:36.329267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 21/42] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Sep 28 12:00:36 2022] POST /identity/v3/projects => generated 301 bytes in 302 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:00:36.478073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 22/43] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:36.562565 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11933c9d-e096-4d3e-9bde-2f202c531ad1 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:36.667701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11933c9d-e096-4d3e-9bde-2f202c531ad1 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:36.702027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3d6e6c-4084-4e86-b2f1-155862386ac9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:36.702975 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3d6e6c-4084-4e86-b2f1-155862386ac9 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:36.703481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3d6e6c-4084-4e86-b2f1-155862386ac9 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:36.703974 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3d6e6c-4084-4e86-b2f1-155862386ac9 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:36.704939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3d6e6c-4084-4e86-b2f1-155862386ac9 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:36.706541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3d6e6c-4084-4e86-b2f1-155862386ac9 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:36.718209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:36.725520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 22/44] 10.222.0.26 () {60 vars in 1131 bytes} [Wed Sep 28 12:00:36 2022] POST /identity/v3/roles => generated 223 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:00:36.748712 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4072ff-7f00-4bce-ae9b-b4769a98eb70 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:36.859281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4072ff-7f00-4bce-ae9b-b4769a98eb70 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:36.861742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:36.899521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b675b7f-1816-4338-b344-4c361c918260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:36.900180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b675b7f-1816-4338-b344-4c361c918260 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:36.900482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b675b7f-1816-4338-b344-4c361c918260 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:36.900758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b675b7f-1816-4338-b344-4c361c918260 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:36.901885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b675b7f-1816-4338-b344-4c361c918260 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:36.903360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b675b7f-1816-4338-b344-4c361c918260 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:36.904957 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 23/45] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:00:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:36.910524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3d6e6c-4084-4e86-b2f1-155862386ac9 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:37.014441 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3d6e6c-4084-4e86-b2f1-155862386ac9 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:37.050160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c29760-4d83-4649-ad59-0f016b2555fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:37.050793 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c29760-4d83-4649-ad59-0f016b2555fe None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:37.051074 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c29760-4d83-4649-ad59-0f016b2555fe None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:37.051349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c29760-4d83-4649-ad59-0f016b2555fe None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:37.052033 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c29760-4d83-4649-ad59-0f016b2555fe None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:37.053557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c29760-4d83-4649-ad59-0f016b2555fe None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:37.081024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 23/46] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Sep 28 12:00:36 2022] POST /identity/v3/projects => generated 312 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:00:38.322215 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677f69aa-f504-4183-aeb6-8d2088611218 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:38.322765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677f69aa-f504-4183-aeb6-8d2088611218 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:38.322765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677f69aa-f504-4183-aeb6-8d2088611218 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:38.324203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 24/47] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:38.329531 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef8491a-8277-437f-a3b2-8e54d12f416d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:38.329954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef8491a-8277-437f-a3b2-8e54d12f416d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:38.330096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef8491a-8277-437f-a3b2-8e54d12f416d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:38.456388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-cef8491a-8277-437f-a3b2-8e54d12f416d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:38.457905 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-cef8491a-8277-437f-a3b2-8e54d12f416d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:38.507616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:38.528834 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 24/48] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:38.799731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677f69aa-f504-4183-aeb6-8d2088611218 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:38.898873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677f69aa-f504-4183-aeb6-8d2088611218 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:38.932981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b1f5a5-ebc7-464c-9b29-32f4703d69bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:38.933603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b1f5a5-ebc7-464c-9b29-32f4703d69bd None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:38.933887 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b1f5a5-ebc7-464c-9b29-32f4703d69bd None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:38.934163 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b1f5a5-ebc7-464c-9b29-32f4703d69bd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:38.944076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b1f5a5-ebc7-464c-9b29-32f4703d69bd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-e9b1f5a5-ebc7-464c-9b29-32f4703d69bd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:38.948971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:38.949785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:38.950513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:38.950513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:38.950513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:38.950513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:38.950513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:38.950513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:38.950513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:38.950513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:38.950513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:38.950513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:38.950513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:00:38.950513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:00:38.950513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 25/49] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:00:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:38.955856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef8491a-8277-437f-a3b2-8e54d12f416d None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:39.052720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef8491a-8277-437f-a3b2-8e54d12f416d None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:39.086387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe51f6e7-24d0-4aae-b0b6-466ce3251cd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:39.087274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe51f6e7-24d0-4aae-b0b6-466ce3251cd8 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:39.087566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe51f6e7-24d0-4aae-b0b6-466ce3251cd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:39.087844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe51f6e7-24d0-4aae-b0b6-466ce3251cd8 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:39.088589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe51f6e7-24d0-4aae-b0b6-466ce3251cd8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:39.088902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe51f6e7-24d0-4aae-b0b6-466ce3251cd8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:39.090351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe51f6e7-24d0-4aae-b0b6-466ce3251cd8 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:39.099810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 25/50] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:00:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:39.107222 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b1f5a5-ebc7-464c-9b29-32f4703d69bd None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:39.141122 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:39.207770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f6d3f3-8bd6-4671-824e-66bfd621d9c0 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:39.208054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f6d3f3-8bd6-4671-824e-66bfd621d9c0 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:39.208355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f6d3f3-8bd6-4671-824e-66bfd621d9c0 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:39.208552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b1f5a5-ebc7-464c-9b29-32f4703d69bd None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:39.209692 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 26/51] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:00:39.215321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa1a605-0906-4cd2-b75d-560c09a6923e None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:39.215611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa1a605-0906-4cd2-b75d-560c09a6923e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:39.215881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa1a605-0906-4cd2-b75d-560c09a6923e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:39.233731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-1aa1a605-0906-4cd2-b75d-560c09a6923e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:39.234925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1aa1a605-0906-4cd2-b75d-560c09a6923e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:39.242834 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9edb8b-af1d-4930-a6a8-181f599d6984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:39.243454 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9edb8b-af1d-4930-a6a8-181f599d6984 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:39.243733 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9edb8b-af1d-4930-a6a8-181f599d6984 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:39.244008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9edb8b-af1d-4930-a6a8-181f599d6984 None admin] PATH_INFO: `/v3/users/f82a94e2366b45ff82ac58061224a3a2` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:39.245155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9edb8b-af1d-4930-a6a8-181f599d6984 None admin] RBAC: Authorizing `identity:get_user(user_id=f82a94e2366b45ff82ac58061224a3a2)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:39.246606 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9edb8b-af1d-4930-a6a8-181f599d6984 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:39.248167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 26/52] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:00:39 2022] GET /identity/v3/users/f82a94e2366b45ff82ac58061224a3a2 => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:39.253406 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9edb8b-af1d-4930-a6a8-181f599d6984 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:39.269144 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:39.301953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 27/53] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:00:39.348210 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:39.351880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9edb8b-af1d-4930-a6a8-181f599d6984 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:39.384746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2033520-327f-4af2-bddd-697b2263ba72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:39.385372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2033520-327f-4af2-bddd-697b2263ba72 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:39.385660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2033520-327f-4af2-bddd-697b2263ba72 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:39.385941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2033520-327f-4af2-bddd-697b2263ba72 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:39.387090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2033520-327f-4af2-bddd-697b2263ba72 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:39.388593 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2033520-327f-4af2-bddd-697b2263ba72 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:39.390156 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 27/54] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:00:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:00:39.396041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa1a605-0906-4cd2-b75d-560c09a6923e None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:39.493634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa1a605-0906-4cd2-b75d-560c09a6923e None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:39.527373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5824b4e8-ae7d-4442-bea7-b8298760a648 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:39.527980 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5824b4e8-ae7d-4442-bea7-b8298760a648 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:39.528263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5824b4e8-ae7d-4442-bea7-b8298760a648 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:39.528577 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5824b4e8-ae7d-4442-bea7-b8298760a648 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:39.529305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5824b4e8-ae7d-4442-bea7-b8298760a648 None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.domain.id=default, user.id=f82a94e2366b45ff82ac58061224a3a2) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:39.529621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5824b4e8-ae7d-4442-bea7-b8298760a648 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:39.531057 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5824b4e8-ae7d-4442-bea7-b8298760a648 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:39.538714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 28/55] 10.222.0.26 () {56 vars in 1322 bytes} [Wed Sep 28 12:00:39 2022] GET /identity/v3/role_assignments?user.id=f82a94e2366b45ff82ac58061224a3a2&scope.domain.id=default&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 530 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:39.571776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2033520-327f-4af2-bddd-697b2263ba72 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:39.676128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2033520-327f-4af2-bddd-697b2263ba72 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:39.712530 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f26121f-d31a-43fa-89aa-0a219c6b0c24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:39.713167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f26121f-d31a-43fa-89aa-0a219c6b0c24 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:39.713451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f26121f-d31a-43fa-89aa-0a219c6b0c24 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:39.713728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f26121f-d31a-43fa-89aa-0a219c6b0c24 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:39.714849 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f26121f-d31a-43fa-89aa-0a219c6b0c24 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:39.716333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f26121f-d31a-43fa-89aa-0a219c6b0c24 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:39.717897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 28/56] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:00:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:39.723489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5824b4e8-ae7d-4442-bea7-b8298760a648 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:39.760294 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:39.822406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5824b4e8-ae7d-4442-bea7-b8298760a648 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:39.854766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a17975e-c470-45d2-842b-52f2a3601503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:39.855378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a17975e-c470-45d2-842b-52f2a3601503 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:39.855663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a17975e-c470-45d2-842b-52f2a3601503 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:39.855939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a17975e-c470-45d2-842b-52f2a3601503 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:39.856618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a17975e-c470-45d2-842b-52f2a3601503 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:39.858067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a17975e-c470-45d2-842b-52f2a3601503 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:39.884292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 29/57] 10.222.0.26 () {60 vars in 1140 bytes} [Wed Sep 28 12:00:39 2022] POST /identity/v3/projects => generated 298 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:00:42.000496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ede88f2-27bf-4406-85e5-ff5ec909d7ee None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:42.001019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ede88f2-27bf-4406-85e5-ff5ec909d7ee None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:42.001019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ede88f2-27bf-4406-85e5-ff5ec909d7ee None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:42.002398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 29/58] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:42.008017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b4fd6e-80b3-4f4a-baeb-03d414b8f094 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:42.008467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b4fd6e-80b3-4f4a-baeb-03d414b8f094 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:42.008622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b4fd6e-80b3-4f4a-baeb-03d414b8f094 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:42.026842 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-76b4fd6e-80b3-4f4a-baeb-03d414b8f094 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:42.028031 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-76b4fd6e-80b3-4f4a-baeb-03d414b8f094 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:42.091267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 30/59] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:42.358936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ede88f2-27bf-4406-85e5-ff5ec909d7ee None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:42.394456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:42.457537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ede88f2-27bf-4406-85e5-ff5ec909d7ee None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:42.489597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03992474-a427-449a-8e82-ae90904a4cf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:42.490230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03992474-a427-449a-8e82-ae90904a4cf4 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:42.490515 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03992474-a427-449a-8e82-ae90904a4cf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:42.490791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03992474-a427-449a-8e82-ae90904a4cf4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:42.492454 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03992474-a427-449a-8e82-ae90904a4cf4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:42.493907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03992474-a427-449a-8e82-ae90904a4cf4 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:42.495469 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 30/60] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:00:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:42.501160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b4fd6e-80b3-4f4a-baeb-03d414b8f094 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:42.519199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:42.600061 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b4fd6e-80b3-4f4a-baeb-03d414b8f094 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:42.632813 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a33a35-b029-4ad4-b103-9dcf429b3a30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:42.633363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a33a35-b029-4ad4-b103-9dcf429b3a30 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:42.633647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a33a35-b029-4ad4-b103-9dcf429b3a30 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:42.633926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a33a35-b029-4ad4-b103-9dcf429b3a30 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:42.634589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a33a35-b029-4ad4-b103-9dcf429b3a30 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:42.636024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a33a35-b029-4ad4-b103-9dcf429b3a30 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:42.661409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 31/61] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:00:42 2022] POST /identity/v3/users => generated 275 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:00:45.994835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a7077b-89f0-4e9c-84a2-dbf1030ed22d None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:45.995380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a7077b-89f0-4e9c-84a2-dbf1030ed22d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:45.995380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a7077b-89f0-4e9c-84a2-dbf1030ed22d None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:45.996937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 31/62] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:46.011345 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e0f70-35ca-4627-8170-6b4d3211bab5 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:46.011837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e0f70-35ca-4627-8170-6b4d3211bab5 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:46.011925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e0f70-35ca-4627-8170-6b4d3211bab5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:46.036546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-af4e0f70-35ca-4627-8170-6b4d3211bab5 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:46.037770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-af4e0f70-35ca-4627-8170-6b4d3211bab5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:46.093730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e602ed0-9f74-4ca5-94ef-de9175334467 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:46.094016 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e602ed0-9f74-4ca5-94ef-de9175334467 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:46.094294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e602ed0-9f74-4ca5-94ef-de9175334467 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:46.095701 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 32/63] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:00:46.108731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065794e9-ed09-49ed-8e0d-4f5e39c28236 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:46.109440 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065794e9-ed09-49ed-8e0d-4f5e39c28236 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:46.109921 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065794e9-ed09-49ed-8e0d-4f5e39c28236 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:46.140864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-065794e9-ed09-49ed-8e0d-4f5e39c28236 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:46.142060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-065794e9-ed09-49ed-8e0d-4f5e39c28236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:46.145211 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 32/64] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:46.209095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a1f788-3cd5-49c9-b8c0-a3924bc20137 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:46.209785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a1f788-3cd5-49c9-b8c0-a3924bc20137 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:46.210262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a1f788-3cd5-49c9-b8c0-a3924bc20137 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:46.211809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 33/65] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:00:46.221936 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7f263d-7568-451c-9977-40b9dbd0867b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:46.222481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7f263d-7568-451c-9977-40b9dbd0867b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:46.222973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7f263d-7568-451c-9977-40b9dbd0867b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:46.233916 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 33/66] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:00:46.245610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:46.266162 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4d7f263d-7568-451c-9977-40b9dbd0867b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:46.267604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4d7f263d-7568-451c-9977-40b9dbd0867b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:46.376428 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 34/67] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:00:46.445362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762dad9b-a198-4a49-90cb-42f1aaf83175 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:46.445867 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762dad9b-a198-4a49-90cb-42f1aaf83175 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:46.446306 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762dad9b-a198-4a49-90cb-42f1aaf83175 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:46.447825 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 34/68] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:46.460744 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4683fedb-884f-4ace-a99b-17094f5c07d1 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:46.461049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4683fedb-884f-4ace-a99b-17094f5c07d1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:46.461315 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4683fedb-884f-4ace-a99b-17094f5c07d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:46.492646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4683fedb-884f-4ace-a99b-17094f5c07d1 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:46.494407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4683fedb-884f-4ace-a99b-17094f5c07d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:46.528924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762dad9b-a198-4a49-90cb-42f1aaf83175 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:46.607866 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 35/69] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:46.609266 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:46.609512 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4683fedb-884f-4ace-a99b-17094f5c07d1 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:46.688196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:46.734911 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4683fedb-884f-4ace-a99b-17094f5c07d1 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:46.753557 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762dad9b-a198-4a49-90cb-42f1aaf83175 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:46.781018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3560b32-8209-4415-9e93-711207912d49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:46.782117 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3560b32-8209-4415-9e93-711207912d49 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:46.782636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3560b32-8209-4415-9e93-711207912d49 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:46.782976 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3560b32-8209-4415-9e93-711207912d49 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:46.790003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005bef1e-f9a9-48e5-a357-e5726e873bd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:46.790640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005bef1e-f9a9-48e5-a357-e5726e873bd3 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:46.790924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005bef1e-f9a9-48e5-a357-e5726e873bd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:46.791198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005bef1e-f9a9-48e5-a357-e5726e873bd3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:46.794477 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3560b32-8209-4415-9e93-711207912d49 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-c3560b32-8209-4415-9e93-711207912d49 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:46.799907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:46.800774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:46.804966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:46.804966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:46.804966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:46.804966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:46.804966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:46.804966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:46.804966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:46.804966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:46.804966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:46.804966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:46.804966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:00:46.804966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:00:46.804966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 36/70] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:00:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:46.804966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3560b32-8209-4415-9e93-711207912d49 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:46.808897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005bef1e-f9a9-48e5-a357-e5726e873bd3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-005bef1e-f9a9-48e5-a357-e5726e873bd3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:46.813934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:46.814807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:46.815553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:46.815553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:46.815553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:46.815553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:46.815553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:46.815553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:46.815553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:46.815553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:46.815553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:46.815553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:46.815553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:00:46.815553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:00:46.820056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 35/71] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:00:46 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:00:46.821361 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5030a168-31bf-43ec-a590-e2d3165aac6f None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:46.822212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5030a168-31bf-43ec-a590-e2d3165aac6f None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:46.822506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5030a168-31bf-43ec-a590-e2d3165aac6f None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:46.824001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 36/72] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:46.824662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5030a168-31bf-43ec-a590-e2d3165aac6f None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:46.918770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3560b32-8209-4415-9e93-711207912d49 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:46.922760 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:46.931397 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5030a168-31bf-43ec-a590-e2d3165aac6f None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:46.941397 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:46.954636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6374da-5dcf-4f7c-a626-27b249a1882d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:46.955267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6374da-5dcf-4f7c-a626-27b249a1882d None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:46.955552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6374da-5dcf-4f7c-a626-27b249a1882d None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:46.955825 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6374da-5dcf-4f7c-a626-27b249a1882d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:46.966182 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fbd061-73da-4aa9-9382-23875e7ec200 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:46.966623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6374da-5dcf-4f7c-a626-27b249a1882d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:46.966791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fbd061-73da-4aa9-9382-23875e7ec200 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:46.967077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fbd061-73da-4aa9-9382-23875e7ec200 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:46.967358 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fbd061-73da-4aa9-9382-23875e7ec200 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:46.968029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fbd061-73da-4aa9-9382-23875e7ec200 None admin] RBAC: Adding query filter params (name=member) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:46.968364 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fbd061-73da-4aa9-9382-23875e7ec200 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:46.969779 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fbd061-73da-4aa9-9382-23875e7ec200 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-6b6374da-5dcf-4f7c-a626-27b249a1882d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:46.971387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:46.972283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:46.973079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:46.973079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:46.973079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:46.973079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:46.973079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:46.973079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:46.973079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:46.973079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:46.973079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:46.973079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:46.973079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:00:46.973079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:00:46.973079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 37/73] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:00:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:46.974256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6374da-5dcf-4f7c-a626-27b249a1882d None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:46.979943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 37/74] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:00:46 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:46.984551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae99ef2-6c31-45e8-b9ce-4e580c47c891 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:46.984551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae99ef2-6c31-45e8-b9ce-4e580c47c891 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:46.984551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae99ef2-6c31-45e8-b9ce-4e580c47c891 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:46.999465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-7ae99ef2-6c31-45e8-b9ce-4e580c47c891 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:47.000669 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-7ae99ef2-6c31-45e8-b9ce-4e580c47c891 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:47.065114 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 38/75] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:47.065854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae99ef2-6c31-45e8-b9ce-4e580c47c891 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:47.084406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6374da-5dcf-4f7c-a626-27b249a1882d None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:47.123333 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdfb2a2-c097-4417-9f86-d8fc0fcb8b9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:47.123949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdfb2a2-c097-4417-9f86-d8fc0fcb8b9e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:47.124228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdfb2a2-c097-4417-9f86-d8fc0fcb8b9e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:47.124534 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdfb2a2-c097-4417-9f86-d8fc0fcb8b9e None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:47.125226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdfb2a2-c097-4417-9f86-d8fc0fcb8b9e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:47.125605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdfb2a2-c097-4417-9f86-d8fc0fcb8b9e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:47.127093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdfb2a2-c097-4417-9f86-d8fc0fcb8b9e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:47.145269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 38/76] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:00:46 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:47.145269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdfb2a2-c097-4417-9f86-d8fc0fcb8b9e None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:47.161765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:47.177630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae99ef2-6c31-45e8-b9ce-4e580c47c891 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:47.195089 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:47.214407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1836f40-4173-453e-8a4b-702a8c05bcef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:47.215037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1836f40-4173-453e-8a4b-702a8c05bcef None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:47.215315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1836f40-4173-453e-8a4b-702a8c05bcef None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:47.215587 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1836f40-4173-453e-8a4b-702a8c05bcef None admin] PATH_INFO: `/v3/roles/member` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:47.226182 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1836f40-4173-453e-8a4b-702a8c05bcef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-c1836f40-4173-453e-8a4b-702a8c05bcef None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:47.230924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:47.231746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:47.232524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:47.232524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:47.232524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:47.232524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:47.232524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:47.232524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:47.232524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:47.232524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:47.232524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:47.232524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:47.232524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:00:47.232524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:00:47.232524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 39/77] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:00:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:47.233154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1836f40-4173-453e-8a4b-702a8c05bcef None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:47.258375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdfb2a2-c097-4417-9f86-d8fc0fcb8b9e None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:47.292496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f99204-87f8-430a-b839-e1f2e0f919ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:47.293102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f99204-87f8-430a-b839-e1f2e0f919ce None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:47.293384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f99204-87f8-430a-b839-e1f2e0f919ce None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:47.293658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f99204-87f8-430a-b839-e1f2e0f919ce None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:47.294372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f99204-87f8-430a-b839-e1f2e0f919ce None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:47.294680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f99204-87f8-430a-b839-e1f2e0f919ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:47.296044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f99204-87f8-430a-b839-e1f2e0f919ce None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:47.303988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 39/78] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:00:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:47.304821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f99204-87f8-430a-b839-e1f2e0f919ce None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:47.339550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1836f40-4173-453e-8a4b-702a8c05bcef None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:47.372908 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d7917e-ce6e-43ca-b7e6-ae0faa01a54e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:47.373531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d7917e-ce6e-43ca-b7e6-ae0faa01a54e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:47.373811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d7917e-ce6e-43ca-b7e6-ae0faa01a54e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:47.374083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d7917e-ce6e-43ca-b7e6-ae0faa01a54e None admin] PATH_INFO: `/v3/users/e481e627a19f4fcaa9af5c5635ce4374` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:47.387611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d7917e-ce6e-43ca-b7e6-ae0faa01a54e None admin] RBAC: Authorizing `identity:get_user(user_id=e481e627a19f4fcaa9af5c5635ce4374)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:47.389119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d7917e-ce6e-43ca-b7e6-ae0faa01a54e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:47.390850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 40/79] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:00:47 2022] GET /identity/v3/users/e481e627a19f4fcaa9af5c5635ce4374 => generated 275 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:47.391608 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d7917e-ce6e-43ca-b7e6-ae0faa01a54e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:47.395646 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:47.408794 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f99204-87f8-430a-b839-e1f2e0f919ce None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:47.417481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:47.442779 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700e8280-66eb-43d9-ace5-75c85fd7f5e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:47.443412 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700e8280-66eb-43d9-ace5-75c85fd7f5e0 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:47.443691 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700e8280-66eb-43d9-ace5-75c85fd7f5e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:47.444022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700e8280-66eb-43d9-ace5-75c85fd7f5e0 None admin] PATH_INFO: `/v3/users/e481e627a19f4fcaa9af5c5635ce4374` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:47.445375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700e8280-66eb-43d9-ace5-75c85fd7f5e0 None admin] RBAC: Authorizing `identity:get_user(user_id=e481e627a19f4fcaa9af5c5635ce4374)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:47.446797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700e8280-66eb-43d9-ace5-75c85fd7f5e0 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:47.448378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 40/80] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:00:47 2022] GET /identity/v3/users/e481e627a19f4fcaa9af5c5635ce4374 => generated 275 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:47.449132 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700e8280-66eb-43d9-ace5-75c85fd7f5e0 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:47.494971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d7917e-ce6e-43ca-b7e6-ae0faa01a54e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:47.529841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d96486-d150-42b0-98b3-fba72429cf0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:47.530448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d96486-d150-42b0-98b3-fba72429cf0a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:47.530729 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d96486-d150-42b0-98b3-fba72429cf0a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:47.531005 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d96486-d150-42b0-98b3-fba72429cf0a None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:47.531676 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d96486-d150-42b0-98b3-fba72429cf0a None admin] RBAC: Adding query filter params (name=member) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:47.531982 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d96486-d150-42b0-98b3-fba72429cf0a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:47.533396 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d96486-d150-42b0-98b3-fba72429cf0a None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:47.541922 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 41/81] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:00:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:47.542714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d96486-d150-42b0-98b3-fba72429cf0a None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:47.555887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700e8280-66eb-43d9-ace5-75c85fd7f5e0 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:47.589830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a0376b-3f09-4cc2-964b-06a379c5ee1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:47.590391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0376b-3f09-4cc2-964b-06a379c5ee1e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:47.590672 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0376b-3f09-4cc2-964b-06a379c5ee1e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:47.590949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0376b-3f09-4cc2-964b-06a379c5ee1e None admin] PATH_INFO: `/v3/users/f82a94e2366b45ff82ac58061224a3a2` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:47.592081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a0376b-3f09-4cc2-964b-06a379c5ee1e None admin] RBAC: Authorizing `identity:get_user(user_id=f82a94e2366b45ff82ac58061224a3a2)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:47.593579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a0376b-3f09-4cc2-964b-06a379c5ee1e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:47.595109 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 41/82] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:00:47 2022] GET /identity/v3/users/f82a94e2366b45ff82ac58061224a3a2 => generated 247 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:47.597183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a0376b-3f09-4cc2-964b-06a379c5ee1e None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:47.606500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:47.640502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d96486-d150-42b0-98b3-fba72429cf0a None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:47.641217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:47.675221 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1414f5da-d8dc-4c13-a561-7797bc51372f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:47.675839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1414f5da-d8dc-4c13-a561-7797bc51372f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:47.676121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1414f5da-d8dc-4c13-a561-7797bc51372f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:47.676428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1414f5da-d8dc-4c13-a561-7797bc51372f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:47.677543 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1414f5da-d8dc-4c13-a561-7797bc51372f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:47.679540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1414f5da-d8dc-4c13-a561-7797bc51372f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:47.681120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 42/83] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:00:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:47.681884 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1414f5da-d8dc-4c13-a561-7797bc51372f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:47.699407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a0376b-3f09-4cc2-964b-06a379c5ee1e None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:47.732240 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30a0655-4a43-4681-80f7-e1bf1da58299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:47.732875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30a0655-4a43-4681-80f7-e1bf1da58299 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:47.733157 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30a0655-4a43-4681-80f7-e1bf1da58299 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:47.733434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30a0655-4a43-4681-80f7-e1bf1da58299 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:47.778377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1414f5da-d8dc-4c13-a561-7797bc51372f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:47.802016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30a0655-4a43-4681-80f7-e1bf1da58299 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:47.804887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30a0655-4a43-4681-80f7-e1bf1da58299 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:47.813528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a95b2-1618-4252-9fb3-d796429bc722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:47.814572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a95b2-1618-4252-9fb3-d796429bc722 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:47.815062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a95b2-1618-4252-9fb3-d796429bc722 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:47.815551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a95b2-1618-4252-9fb3-d796429bc722 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:47.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:47.853910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:47.894383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73a95b2-1618-4252-9fb3-d796429bc722 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:47.895933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73a95b2-1618-4252-9fb3-d796429bc722 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:47.908048 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 42/84] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:00:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:00:47.909073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30a0655-4a43-4681-80f7-e1bf1da58299 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:47.992597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 43/85] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:00:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:00:47.993525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a95b2-1618-4252-9fb3-d796429bc722 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:48.014793 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30a0655-4a43-4681-80f7-e1bf1da58299 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:48.026425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:48.047932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92580814-6439-40e8-bd49-0d1d5e3e9c0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:48.048590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92580814-6439-40e8-bd49-0d1d5e3e9c0a None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:48.048873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92580814-6439-40e8-bd49-0d1d5e3e9c0a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:48.049159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92580814-6439-40e8-bd49-0d1d5e3e9c0a None admin] PATH_INFO: `/v3/users/e481e627a19f4fcaa9af5c5635ce4374` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:48.050290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92580814-6439-40e8-bd49-0d1d5e3e9c0a None admin] RBAC: Authorizing `identity:get_user(user_id=e481e627a19f4fcaa9af5c5635ce4374)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:48.051732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92580814-6439-40e8-bd49-0d1d5e3e9c0a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:48.053386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 43/86] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:00:47 2022] GET /identity/v3/users/e481e627a19f4fcaa9af5c5635ce4374 => generated 275 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:48.054209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92580814-6439-40e8-bd49-0d1d5e3e9c0a None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:48.070592 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:48.095043 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a95b2-1618-4252-9fb3-d796429bc722 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:48.128013 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d75f98-6d63-4d98-8c07-25872510eedd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:48.128581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d75f98-6d63-4d98-8c07-25872510eedd None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:48.128861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d75f98-6d63-4d98-8c07-25872510eedd None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:48.129130 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d75f98-6d63-4d98-8c07-25872510eedd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:48.157342 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92580814-6439-40e8-bd49-0d1d5e3e9c0a None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:48.192597 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec1b1fc-a70f-4775-8169-05edbdc82eda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:48.193206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec1b1fc-a70f-4775-8169-05edbdc82eda None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:48.193488 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec1b1fc-a70f-4775-8169-05edbdc82eda None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:48.193762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec1b1fc-a70f-4775-8169-05edbdc82eda None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:48.194977 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec1b1fc-a70f-4775-8169-05edbdc82eda None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:48.196460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec1b1fc-a70f-4775-8169-05edbdc82eda None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:48.201944 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d75f98-6d63-4d98-8c07-25872510eedd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:48.203411 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d75f98-6d63-4d98-8c07-25872510eedd None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:48.218124 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 44/87] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:00:48 2022] POST /identity/v3/users => generated 289 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:00:48.221496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec1b1fc-a70f-4775-8169-05edbdc82eda None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:48.238689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:48.306669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:48.315555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 44/88] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:00:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:00:48.316497 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d75f98-6d63-4d98-8c07-25872510eedd None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:48.354321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec1b1fc-a70f-4775-8169-05edbdc82eda None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:48.395231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa34f9e-5f93-4603-96fa-ea96ee642df3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:48.395790 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa34f9e-5f93-4603-96fa-ea96ee642df3 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:48.396072 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa34f9e-5f93-4603-96fa-ea96ee642df3 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:48.396374 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa34f9e-5f93-4603-96fa-ea96ee642df3 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:48.398065 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa34f9e-5f93-4603-96fa-ea96ee642df3 None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:48.399505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa34f9e-5f93-4603-96fa-ea96ee642df3 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:48.401236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 45/89] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:00:48 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:48.406770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa34f9e-5f93-4603-96fa-ea96ee642df3 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:48.430857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d75f98-6d63-4d98-8c07-25872510eedd None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:48.469959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50b69ac-3609-4d30-9f66-8e0d13e34967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:48.470612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50b69ac-3609-4d30-9f66-8e0d13e34967 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:48.470926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50b69ac-3609-4d30-9f66-8e0d13e34967 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:48.471164 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50b69ac-3609-4d30-9f66-8e0d13e34967 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:48.472589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50b69ac-3609-4d30-9f66-8e0d13e34967 None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:48.473875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50b69ac-3609-4d30-9f66-8e0d13e34967 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:48.476456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 45/90] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:00:48 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:48.477294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50b69ac-3609-4d30-9f66-8e0d13e34967 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:48.479992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:48.536255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa34f9e-5f93-4603-96fa-ea96ee642df3 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:48.574956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ae3157-1bd4-43b2-8f8b-03936c9e4f16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:48.575926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ae3157-1bd4-43b2-8f8b-03936c9e4f16 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:48.575926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ae3157-1bd4-43b2-8f8b-03936c9e4f16 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:48.576146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ae3157-1bd4-43b2-8f8b-03936c9e4f16 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:48.582139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:48.609109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50b69ac-3609-4d30-9f66-8e0d13e34967 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:48.651178 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1dfe99-8878-4861-b1aa-9b4b0c6c8fc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:48.652110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1dfe99-8878-4861-b1aa-9b4b0c6c8fc9 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:48.652583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1dfe99-8878-4861-b1aa-9b4b0c6c8fc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:48.653028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1dfe99-8878-4861-b1aa-9b4b0c6c8fc9 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:48.654372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1dfe99-8878-4861-b1aa-9b4b0c6c8fc9 None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:48.656023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1dfe99-8878-4861-b1aa-9b4b0c6c8fc9 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:48.657847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 46/91] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:00:48 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:48.658830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1dfe99-8878-4861-b1aa-9b4b0c6c8fc9 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:48.668541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ae3157-1bd4-43b2-8f8b-03936c9e4f16 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:48.670382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ae3157-1bd4-43b2-8f8b-03936c9e4f16 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:48.741445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:48.779784 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1dfe99-8878-4861-b1aa-9b4b0c6c8fc9 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:48.797345 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 46/92] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:00:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:00:48.798232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ae3157-1bd4-43b2-8f8b-03936c9e4f16 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:48.822742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7f0333-b726-4efa-93c0-9bf85dd7359e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:48.823714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7f0333-b726-4efa-93c0-9bf85dd7359e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:48.824216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7f0333-b726-4efa-93c0-9bf85dd7359e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:48.824717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7f0333-b726-4efa-93c0-9bf85dd7359e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:48.825830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7f0333-b726-4efa-93c0-9bf85dd7359e None admin] RBAC: Adding query filter params (role.id=615e83fab2504b6298db3efaa7c15750, scope.project.id=47fa4785636148129d33fffc2096cd4b, user.id=e481e627a19f4fcaa9af5c5635ce4374) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:48.826333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7f0333-b726-4efa-93c0-9bf85dd7359e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:48.828072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7f0333-b726-4efa-93c0-9bf85dd7359e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:48.840810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:48.841605 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 47/93] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:00:48 2022] GET /identity/v3/role_assignments?user.id=e481e627a19f4fcaa9af5c5635ce4374&scope.project.id=47fa4785636148129d33fffc2096cd4b&role.id=615e83fab2504b6298db3efaa7c15750 => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:48.842792 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7f0333-b726-4efa-93c0-9bf85dd7359e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:48.981363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ae3157-1bd4-43b2-8f8b-03936c9e4f16 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:49.043149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7f0333-b726-4efa-93c0-9bf85dd7359e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:49.055122 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c571708-4dee-40cb-a78f-85cf2cd799b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:49.055740 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c571708-4dee-40cb-a78f-85cf2cd799b7 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:49.056027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c571708-4dee-40cb-a78f-85cf2cd799b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:49.056354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c571708-4dee-40cb-a78f-85cf2cd799b7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:49.057199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c571708-4dee-40cb-a78f-85cf2cd799b7 None admin] RBAC: Adding query filter params (role.id=a3902a185ac144fb8e24f4e9feb039ed, scope.project.id=47fa4785636148129d33fffc2096cd4b, user.id=e481e627a19f4fcaa9af5c5635ce4374) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:49.057514 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c571708-4dee-40cb-a78f-85cf2cd799b7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:49.059080 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c571708-4dee-40cb-a78f-85cf2cd799b7 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:49.065276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 47/94] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:00:48 2022] GET /identity/v3/role_assignments?user.id=e481e627a19f4fcaa9af5c5635ce4374&scope.project.id=47fa4785636148129d33fffc2096cd4b&role.id=a3902a185ac144fb8e24f4e9feb039ed => generated 259 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:49.065882 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c571708-4dee-40cb-a78f-85cf2cd799b7 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:49.118801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f05a6d7-7f33-4d77-9440-57407dad3ebb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:49.119429 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05a6d7-7f33-4d77-9440-57407dad3ebb None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:49.119718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05a6d7-7f33-4d77-9440-57407dad3ebb None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:49.119993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05a6d7-7f33-4d77-9440-57407dad3ebb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:49.120775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f05a6d7-7f33-4d77-9440-57407dad3ebb None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=47fa4785636148129d33fffc2096cd4b, user.id=f82a94e2366b45ff82ac58061224a3a2) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:49.121370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f05a6d7-7f33-4d77-9440-57407dad3ebb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:49.122866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f05a6d7-7f33-4d77-9440-57407dad3ebb None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:49.125252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:49.135193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 48/95] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:00:48 2022] GET /identity/v3/role_assignments?user.id=f82a94e2366b45ff82ac58061224a3a2&scope.project.id=47fa4785636148129d33fffc2096cd4b&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 259 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:49.237826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:49.241752 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c571708-4dee-40cb-a78f-85cf2cd799b7 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:49.288285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721a0e2a-9c88-452f-9186-355e75bdcf23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:49.288975 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721a0e2a-9c88-452f-9186-355e75bdcf23 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:49.289260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721a0e2a-9c88-452f-9186-355e75bdcf23 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:49.289541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721a0e2a-9c88-452f-9186-355e75bdcf23 None admin] PATH_INFO: `/v3/projects/b8dbeed180024b75b872ce3151dcc7f0` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:49.293890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721a0e2a-9c88-452f-9186-355e75bdcf23 None admin] RBAC: Authorizing `identity:get_project(project_id=b8dbeed180024b75b872ce3151dcc7f0)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:49.295446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721a0e2a-9c88-452f-9186-355e75bdcf23 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:49.297512 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 48/96] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:00:49 2022] GET /identity/v3/projects/b8dbeed180024b75b872ce3151dcc7f0 => generated 312 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:49.305569 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f05a6d7-7f33-4d77-9440-57407dad3ebb None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:49.477673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f05a6d7-7f33-4d77-9440-57407dad3ebb None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:49.526665 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531246ea-2864-4a70-9b26-2fe1453c0065 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:49.527288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531246ea-2864-4a70-9b26-2fe1453c0065 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:49.527564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531246ea-2864-4a70-9b26-2fe1453c0065 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:49.527844 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531246ea-2864-4a70-9b26-2fe1453c0065 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:49.528611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531246ea-2864-4a70-9b26-2fe1453c0065 None admin] RBAC: Adding query filter params (role.id=615e83fab2504b6298db3efaa7c15750, scope.project.id=b8dbeed180024b75b872ce3151dcc7f0, user.id=e481e627a19f4fcaa9af5c5635ce4374) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:49.528925 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531246ea-2864-4a70-9b26-2fe1453c0065 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:49.530375 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531246ea-2864-4a70-9b26-2fe1453c0065 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:49.536164 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 49/97] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:00:49 2022] GET /identity/v3/role_assignments?user.id=e481e627a19f4fcaa9af5c5635ce4374&scope.project.id=b8dbeed180024b75b872ce3151dcc7f0&role.id=615e83fab2504b6298db3efaa7c15750 => generated 259 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:51.168187 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a23542-c1b4-40cf-a89e-aa71469c4602 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:51.168731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a23542-c1b4-40cf-a89e-aa71469c4602 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:51.168804 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a23542-c1b4-40cf-a89e-aa71469c4602 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:51.170403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 49/98] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:51.175321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f6a809-b161-4072-b1ea-d72929339c87 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:51.175710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f6a809-b161-4072-b1ea-d72929339c87 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:51.175893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f6a809-b161-4072-b1ea-d72929339c87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:51.201050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-54f6a809-b161-4072-b1ea-d72929339c87 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:51.202281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-54f6a809-b161-4072-b1ea-d72929339c87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:51.245337 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e39b5ef-d3a7-42e4-89f3-2818835a6632 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:51.245654 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e39b5ef-d3a7-42e4-89f3-2818835a6632 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:51.245896 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e39b5ef-d3a7-42e4-89f3-2818835a6632 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:51.247235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 50/99] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:51.252229 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c816093e-10dd-4105-92a5-f275e852b372 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:51.252551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c816093e-10dd-4105-92a5-f275e852b372 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:51.252833 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c816093e-10dd-4105-92a5-f275e852b372 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:51.279375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-c816093e-10dd-4105-92a5-f275e852b372 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:51.280608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c816093e-10dd-4105-92a5-f275e852b372 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:51.314183 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:51.339900 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 50/100] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:51.384861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 51/101] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:00:51.696778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f6a809-b161-4072-b1ea-d72929339c87 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:51.793912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a823bac3-3bf6-4b07-8636-e57b7e982851 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:51.794226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a823bac3-3bf6-4b07-8636-e57b7e982851 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:51.794480 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a823bac3-3bf6-4b07-8636-e57b7e982851 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:51.795953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 52/102] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:00:51.801441 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13563d79-4c5f-47aa-a885-ab6e026cde44 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:51.801731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13563d79-4c5f-47aa-a885-ab6e026cde44 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:51.802009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13563d79-4c5f-47aa-a885-ab6e026cde44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:51.825924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-13563d79-4c5f-47aa-a885-ab6e026cde44 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:51.827125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-13563d79-4c5f-47aa-a885-ab6e026cde44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:51.836564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f6a809-b161-4072-b1ea-d72929339c87 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:51.898175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c00773-ed7b-47a3-83ff-584e9190c785 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:51.898816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c00773-ed7b-47a3-83ff-584e9190c785 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:51.899102 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c00773-ed7b-47a3-83ff-584e9190c785 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:51.899378 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c00773-ed7b-47a3-83ff-584e9190c785 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:51.906000 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c00773-ed7b-47a3-83ff-584e9190c785 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:51.907475 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c00773-ed7b-47a3-83ff-584e9190c785 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:51.909279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 51/103] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:00:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:51.914768 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c00773-ed7b-47a3-83ff-584e9190c785 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:51.961408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 53/104] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:00:51.962352 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13563d79-4c5f-47aa-a885-ab6e026cde44 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:52.017439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:52.040090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:52.110810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c00773-ed7b-47a3-83ff-584e9190c785 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:52.155710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13563d79-4c5f-47aa-a885-ab6e026cde44 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:52.193508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ddc509-dafa-4426-b68e-b78aca15a42f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:52.194953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ddc509-dafa-4426-b68e-b78aca15a42f None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:52.195384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ddc509-dafa-4426-b68e-b78aca15a42f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:52.195803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ddc509-dafa-4426-b68e-b78aca15a42f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:52.201040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512b5429-25ed-4409-9cd2-d0ab5188794d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:52.202236 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512b5429-25ed-4409-9cd2-d0ab5188794d None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:52.202663 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512b5429-25ed-4409-9cd2-d0ab5188794d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:52.203079 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512b5429-25ed-4409-9cd2-d0ab5188794d None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:52.203888 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512b5429-25ed-4409-9cd2-d0ab5188794d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:52.205534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512b5429-25ed-4409-9cd2-d0ab5188794d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:52.207442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ddc509-dafa-4426-b68e-b78aca15a42f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-c7ddc509-dafa-4426-b68e-b78aca15a42f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:52.213453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:52.214473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:52.215399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:52.215399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:52.215399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:52.215399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:52.215399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:52.215399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:52.215399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:52.215399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:52.215399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:52.215399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:52.215399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:00:52.215399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:00:52.217230 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 54/105] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:00:51 2022] GET /identity/v3/roles/reader => generated 84 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:52.225846 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ddc509-dafa-4426-b68e-b78aca15a42f None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:52.259948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 52/106] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Sep 28 12:00:51 2022] POST /identity/v3/projects => generated 302 bytes in 345 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:00:52.329891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2928b1b1-60c2-4333-8c29-8f8b0c4a22b4 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:52.330177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2928b1b1-60c2-4333-8c29-8f8b0c4a22b4 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:52.330458 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2928b1b1-60c2-4333-8c29-8f8b0c4a22b4 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:52.331844 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 53/107] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:00:52.338506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea6006d-934e-40f2-bf98-5de72b337730 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:52.339050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea6006d-934e-40f2-bf98-5de72b337730 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:52.339474 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea6006d-934e-40f2-bf98-5de72b337730 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:52.376807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:52.377617 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-9ea6006d-934e-40f2-bf98-5de72b337730 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:52.378794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9ea6006d-934e-40f2-bf98-5de72b337730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:52.392355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ddc509-dafa-4426-b68e-b78aca15a42f None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:52.448162 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b43873-8a71-48db-8863-ad41653ffde4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:52.449238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b43873-8a71-48db-8863-ad41653ffde4 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:52.449524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b43873-8a71-48db-8863-ad41653ffde4 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:52.449803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b43873-8a71-48db-8863-ad41653ffde4 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:52.450501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b43873-8a71-48db-8863-ad41653ffde4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:52.450801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b43873-8a71-48db-8863-ad41653ffde4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:52.452216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b43873-8a71-48db-8863-ad41653ffde4 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:52.468613 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 55/108] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:00:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:52.476703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b43873-8a71-48db-8863-ad41653ffde4 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:52.544698 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 54/109] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:00:52.545942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea6006d-934e-40f2-bf98-5de72b337730 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:52.596925 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:52.676377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b43873-8a71-48db-8863-ad41653ffde4 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:52.702857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea6006d-934e-40f2-bf98-5de72b337730 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:52.744667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8041bbf4-8c21-41b5-a392-84f0574d83dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:52.745310 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8041bbf4-8c21-41b5-a392-84f0574d83dc None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:52.745586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8041bbf4-8c21-41b5-a392-84f0574d83dc None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:52.746053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8041bbf4-8c21-41b5-a392-84f0574d83dc None admin] PATH_INFO: `/v3/roles/member` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:52.751156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6b6825-e5f4-4fad-b5f1-9a319d8e8c95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:52.752077 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6b6825-e5f4-4fad-b5f1-9a319d8e8c95 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:52.752477 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6b6825-e5f4-4fad-b5f1-9a319d8e8c95 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:52.752674 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6b6825-e5f4-4fad-b5f1-9a319d8e8c95 None admin] PATH_INFO: `/v3/users/dfc58f7c89394e26a4110f493c6fa02d` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:52.758780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8041bbf4-8c21-41b5-a392-84f0574d83dc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-8041bbf4-8c21-41b5-a392-84f0574d83dc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:52.763974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:52.764934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:52.765667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:52.765667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:52.765667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:52.765667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:52.765667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:52.765667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:52.765667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:52.765667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:52.765667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:52.765667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:52.765667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:00:52.765667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:00:52.767863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 55/110] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:00:52 2022] GET /identity/v3/roles/member => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:52.769738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315a95be-03a7-4af3-b5ce-67efa4784926 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:52.770333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315a95be-03a7-4af3-b5ce-67efa4784926 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:52.770628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6b6825-e5f4-4fad-b5f1-9a319d8e8c95 None admin] RBAC: Authorizing `identity:get_user(user_id=dfc58f7c89394e26a4110f493c6fa02d)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:52.771063 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315a95be-03a7-4af3-b5ce-67efa4784926 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:52.772731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6b6825-e5f4-4fad-b5f1-9a319d8e8c95 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:52.773623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 56/111] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:52.774526 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315a95be-03a7-4af3-b5ce-67efa4784926 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:52.774903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 56/112] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:00:52 2022] GET /identity/v3/users/dfc58f7c89394e26a4110f493c6fa02d => generated 289 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:00:52.780131 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6b6825-e5f4-4fad-b5f1-9a319d8e8c95 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:52.791615 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:52.889108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315a95be-03a7-4af3-b5ce-67efa4784926 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:52.891290 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:52.909321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6b6825-e5f4-4fad-b5f1-9a319d8e8c95 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:52.927694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65344390-a8ad-49a7-ab8b-2d57df94c763 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:52.929217 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65344390-a8ad-49a7-ab8b-2d57df94c763 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:52.929697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65344390-a8ad-49a7-ab8b-2d57df94c763 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:52.930179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65344390-a8ad-49a7-ab8b-2d57df94c763 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:52.931081 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65344390-a8ad-49a7-ab8b-2d57df94c763 None admin] RBAC: Adding query filter params (name=member) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:52.931574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65344390-a8ad-49a7-ab8b-2d57df94c763 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:52.933353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65344390-a8ad-49a7-ab8b-2d57df94c763 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:52.941954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 57/113] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:00:52 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:52.943373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d079eb-4ee4-4950-bd75-45edaa174411 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:52.944006 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d079eb-4ee4-4950-bd75-45edaa174411 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:52.944493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d079eb-4ee4-4950-bd75-45edaa174411 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:52.948405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06aa26b7-8e76-477c-a257-3fc69936c478 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:52.949024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06aa26b7-8e76-477c-a257-3fc69936c478 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:52.949310 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06aa26b7-8e76-477c-a257-3fc69936c478 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:52.949588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06aa26b7-8e76-477c-a257-3fc69936c478 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:52.966886 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-49d079eb-4ee4-4950-bd75-45edaa174411 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:52.969865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-49d079eb-4ee4-4950-bd75-45edaa174411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:53.029090 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06aa26b7-8e76-477c-a257-3fc69936c478 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:53.030556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06aa26b7-8e76-477c-a257-3fc69936c478 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:53.040250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:53.044084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 58/114] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:53.045068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d079eb-4ee4-4950-bd75-45edaa174411 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:53.143420 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:53.155027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d079eb-4ee4-4950-bd75-45edaa174411 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:53.165491 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 57/115] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:00:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 28 12:00:53.166162 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06aa26b7-8e76-477c-a257-3fc69936c478 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:53.201316 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0292c899-349e-4964-8e77-497abd14efc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:53.203475 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0292c899-349e-4964-8e77-497abd14efc2 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:53.203955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0292c899-349e-4964-8e77-497abd14efc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:53.204448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0292c899-349e-4964-8e77-497abd14efc2 None admin] PATH_INFO: `/v3/users/e481e627a19f4fcaa9af5c5635ce4374` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:53.205796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0292c899-349e-4964-8e77-497abd14efc2 None admin] RBAC: Authorizing `identity:get_user(user_id=e481e627a19f4fcaa9af5c5635ce4374)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:53.207990 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0292c899-349e-4964-8e77-497abd14efc2 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:53.210426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 59/116] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:00:53 2022] GET /identity/v3/users/e481e627a19f4fcaa9af5c5635ce4374 => generated 275 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:53.212463 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff20192-2aea-4053-b0d4-2f4fd0652640 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:53.213200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff20192-2aea-4053-b0d4-2f4fd0652640 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:53.213670 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff20192-2aea-4053-b0d4-2f4fd0652640 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:53.215776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 60/117] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:53.216723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff20192-2aea-4053-b0d4-2f4fd0652640 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:53.288694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06aa26b7-8e76-477c-a257-3fc69936c478 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:53.301255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:53.327327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff20192-2aea-4053-b0d4-2f4fd0652640 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:53.330850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd29d03-65bb-4f93-bf75-65b47e15dc06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:53.331480 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd29d03-65bb-4f93-bf75-65b47e15dc06 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:53.331762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd29d03-65bb-4f93-bf75-65b47e15dc06 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:53.332032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd29d03-65bb-4f93-bf75-65b47e15dc06 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:53.343996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd29d03-65bb-4f93-bf75-65b47e15dc06 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-6cd29d03-65bb-4f93-bf75-65b47e15dc06 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:53.349986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:53.350839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:53.351572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:53.351572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:53.351572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:53.351572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:53.351572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:53.351572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:53.351572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:53.351572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:53.351572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:53.351572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:53.351572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:00:53.351572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:00:53.351572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 58/118] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:00:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:53.352175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd29d03-65bb-4f93-bf75-65b47e15dc06 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:53.367607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a01dd2-e30e-42b4-a673-ff09ae8fc670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:53.368237 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a01dd2-e30e-42b4-a673-ff09ae8fc670 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:53.368559 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a01dd2-e30e-42b4-a673-ff09ae8fc670 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:53.369181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a01dd2-e30e-42b4-a673-ff09ae8fc670 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:53.370636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a01dd2-e30e-42b4-a673-ff09ae8fc670 None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:53.372098 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a01dd2-e30e-42b4-a673-ff09ae8fc670 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:53.373758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 61/119] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:00:53 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:53.374847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391f7ce1-b235-4a13-b8b1-ca307802c04e None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:53.375127 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391f7ce1-b235-4a13-b8b1-ca307802c04e None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:53.375397 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391f7ce1-b235-4a13-b8b1-ca307802c04e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:53.394735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-391f7ce1-b235-4a13-b8b1-ca307802c04e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:53.395907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-391f7ce1-b235-4a13-b8b1-ca307802c04e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:53.422175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:53.468118 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 62/120] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:53.469273 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391f7ce1-b235-4a13-b8b1-ca307802c04e None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:53.479380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd29d03-65bb-4f93-bf75-65b47e15dc06 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:53.535645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafa4f6c-5498-4409-8eb2-aeb97d7f52f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:53.537114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafa4f6c-5498-4409-8eb2-aeb97d7f52f4 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:53.537853 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafa4f6c-5498-4409-8eb2-aeb97d7f52f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:53.538305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafa4f6c-5498-4409-8eb2-aeb97d7f52f4 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:53.540361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafa4f6c-5498-4409-8eb2-aeb97d7f52f4 None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:53.542499 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafa4f6c-5498-4409-8eb2-aeb97d7f52f4 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:53.545336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 59/121] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:00:53 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:53.546815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafa4f6c-5498-4409-8eb2-aeb97d7f52f4 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:53.595594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:53.596107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391f7ce1-b235-4a13-b8b1-ca307802c04e None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:53.639775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e85e32-ff73-46d9-b68e-c1864c65f3b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:53.640564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e85e32-ff73-46d9-b68e-c1864c65f3b0 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:53.640928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e85e32-ff73-46d9-b68e-c1864c65f3b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:53.641267 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e85e32-ff73-46d9-b68e-c1864c65f3b0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:53.654632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e85e32-ff73-46d9-b68e-c1864c65f3b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-a1e85e32-ff73-46d9-b68e-c1864c65f3b0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:53.660868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:53.661683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:53.667892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:53.667892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:53.667892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:53.667892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:53.667892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:53.667892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:53.667892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:53.667892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:53.667892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:53.667892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:53.667892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:00:53.667892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:00:53.667892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 63/122] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:00:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:53.667892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e85e32-ff73-46d9-b68e-c1864c65f3b0 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:53.680979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafa4f6c-5498-4409-8eb2-aeb97d7f52f4 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:53.692544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:53.717755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c80926-2c1c-4971-b17a-603c1bd44de1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:53.718430 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c80926-2c1c-4971-b17a-603c1bd44de1 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:53.718656 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c80926-2c1c-4971-b17a-603c1bd44de1 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:53.718930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c80926-2c1c-4971-b17a-603c1bd44de1 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:53.719599 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c80926-2c1c-4971-b17a-603c1bd44de1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:53.720199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c80926-2c1c-4971-b17a-603c1bd44de1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:53.721365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c80926-2c1c-4971-b17a-603c1bd44de1 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:53.730551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 60/123] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:00:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:53.731190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c80926-2c1c-4971-b17a-603c1bd44de1 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:53.784574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e85e32-ff73-46d9-b68e-c1864c65f3b0 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:53.820399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c8b36f-168d-4e9e-a9d4-b9b08eb2f21f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:53.821073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c8b36f-168d-4e9e-a9d4-b9b08eb2f21f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:53.821373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c8b36f-168d-4e9e-a9d4-b9b08eb2f21f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:53.821648 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c8b36f-168d-4e9e-a9d4-b9b08eb2f21f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:53.822376 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c8b36f-168d-4e9e-a9d4-b9b08eb2f21f None admin] RBAC: Adding query filter params (role.id=46cb171417764f9f9710877b8ec7becc, scope.project.id=47fa4785636148129d33fffc2096cd4b, user.id=dfc58f7c89394e26a4110f493c6fa02d) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:53.822686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c8b36f-168d-4e9e-a9d4-b9b08eb2f21f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:53.824136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c8b36f-168d-4e9e-a9d4-b9b08eb2f21f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:53.827636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:53.829797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 64/124] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:00:53 2022] GET /identity/v3/role_assignments?user.id=dfc58f7c89394e26a4110f493c6fa02d&scope.project.id=47fa4785636148129d33fffc2096cd4b&role.id=46cb171417764f9f9710877b8ec7becc => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:53.830412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c8b36f-168d-4e9e-a9d4-b9b08eb2f21f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:53.842981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c80926-2c1c-4971-b17a-603c1bd44de1 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:53.882877 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacf95b3-9d71-422d-bbf3-97fd2c6ede87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:53.883795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacf95b3-9d71-422d-bbf3-97fd2c6ede87 None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:53.884214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacf95b3-9d71-422d-bbf3-97fd2c6ede87 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:53.884668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacf95b3-9d71-422d-bbf3-97fd2c6ede87 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b/users/e481e627a19f4fcaa9af5c5635ce4374/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:53.886960 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacf95b3-9d71-422d-bbf3-97fd2c6ede87 None admin] RBAC: Authorizing `identity:create_grant(project_id=47fa4785636148129d33fffc2096cd4b, user_id=e481e627a19f4fcaa9af5c5635ce4374, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:53.888439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacf95b3-9d71-422d-bbf3-97fd2c6ede87 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:53.906175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 61/125] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:00:53 2022] PUT /identity/v3/projects/47fa4785636148129d33fffc2096cd4b/users/e481e627a19f4fcaa9af5c5635ce4374/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:00:53.913435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacf95b3-9d71-422d-bbf3-97fd2c6ede87 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:53.992856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:54.018027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c8b36f-168d-4e9e-a9d4-b9b08eb2f21f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:54.091591 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1352d3e-e05e-4316-8212-0cfc807fad06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:54.092410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1352d3e-e05e-4316-8212-0cfc807fad06 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:54.092691 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1352d3e-e05e-4316-8212-0cfc807fad06 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:54.092963 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1352d3e-e05e-4316-8212-0cfc807fad06 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:54.093648 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1352d3e-e05e-4316-8212-0cfc807fad06 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:54.093952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1352d3e-e05e-4316-8212-0cfc807fad06 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:54.096584 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1352d3e-e05e-4316-8212-0cfc807fad06 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:54.115119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 65/126] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:00:53 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:54.123136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1352d3e-e05e-4316-8212-0cfc807fad06 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:54.133094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacf95b3-9d71-422d-bbf3-97fd2c6ede87 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:54.176507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0532f81b-0120-4382-860e-d4d10a285eed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:54.177566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0532f81b-0120-4382-860e-d4d10a285eed None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:54.178719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0532f81b-0120-4382-860e-d4d10a285eed None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:54.179169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0532f81b-0120-4382-860e-d4d10a285eed None admin] PATH_INFO: `/v3/users/f82a94e2366b45ff82ac58061224a3a2` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:54.180533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0532f81b-0120-4382-860e-d4d10a285eed None admin] RBAC: Authorizing `identity:get_user(user_id=f82a94e2366b45ff82ac58061224a3a2)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:54.182190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0532f81b-0120-4382-860e-d4d10a285eed None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:54.184069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 62/127] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:00:53 2022] GET /identity/v3/users/f82a94e2366b45ff82ac58061224a3a2 => generated 247 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:54.185353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0532f81b-0120-4382-860e-d4d10a285eed None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:54.218199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:54.253399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1352d3e-e05e-4316-8212-0cfc807fad06 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:54.321642 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f15a9-def4-42c1-bd59-bb5bc69ba481 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:54.322303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f15a9-def4-42c1-bd59-bb5bc69ba481 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:54.322584 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f15a9-def4-42c1-bd59-bb5bc69ba481 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:54.322857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f15a9-def4-42c1-bd59-bb5bc69ba481 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:54.347042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130f15a9-def4-42c1-bd59-bb5bc69ba481 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:54.356176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0532f81b-0120-4382-860e-d4d10a285eed None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-130f15a9-def4-42c1-bd59-bb5bc69ba481 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:54.360072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:54.362097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:54.363951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:54.363951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:54.363951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:54.363951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:54.363951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:54.363951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:54.363951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:54.363951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:54.363951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:54.363951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:54.363951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:00:54.363951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:00:54.368223 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 66/128] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:00:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:54.369142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f15a9-def4-42c1-bd59-bb5bc69ba481 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:54.389503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:54.407605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3873c6f-ebc2-498e-b4b6-cecb17861bd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:54.408244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3873c6f-ebc2-498e-b4b6-cecb17861bd2 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:54.408628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3873c6f-ebc2-498e-b4b6-cecb17861bd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:54.408903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3873c6f-ebc2-498e-b4b6-cecb17861bd2 None admin] PATH_INFO: `/v3/users/e481e627a19f4fcaa9af5c5635ce4374` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:54.410048 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3873c6f-ebc2-498e-b4b6-cecb17861bd2 None admin] RBAC: Authorizing `identity:get_user(user_id=e481e627a19f4fcaa9af5c5635ce4374)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:54.411480 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3873c6f-ebc2-498e-b4b6-cecb17861bd2 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:54.413127 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 63/129] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:00:54 2022] GET /identity/v3/users/e481e627a19f4fcaa9af5c5635ce4374 => generated 275 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:54.413899 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3873c6f-ebc2-498e-b4b6-cecb17861bd2 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:54.519389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:54.529009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3873c6f-ebc2-498e-b4b6-cecb17861bd2 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:54.551885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f15a9-def4-42c1-bd59-bb5bc69ba481 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:54.563432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15e790a-c786-4d51-b94e-2d0ee83a495a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:54.564042 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15e790a-c786-4d51-b94e-2d0ee83a495a None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:54.564335 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15e790a-c786-4d51-b94e-2d0ee83a495a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:54.564635 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15e790a-c786-4d51-b94e-2d0ee83a495a None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:54.565305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15e790a-c786-4d51-b94e-2d0ee83a495a None admin] RBAC: Adding query filter params (name=member) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:54.565609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15e790a-c786-4d51-b94e-2d0ee83a495a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:54.566979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15e790a-c786-4d51-b94e-2d0ee83a495a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:54.588224 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 64/130] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:00:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:54.589105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15e790a-c786-4d51-b94e-2d0ee83a495a None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:54.596031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d0480e-0524-4a17-8077-70fcd5310f7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:54.596707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d0480e-0524-4a17-8077-70fcd5310f7e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:54.596997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d0480e-0524-4a17-8077-70fcd5310f7e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:54.597272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d0480e-0524-4a17-8077-70fcd5310f7e None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:54.598429 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d0480e-0524-4a17-8077-70fcd5310f7e None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:54.599926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d0480e-0524-4a17-8077-70fcd5310f7e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:54.606628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 67/131] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:00:54 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:54.607480 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d0480e-0524-4a17-8077-70fcd5310f7e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:54.700112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:54.719093 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d0480e-0524-4a17-8077-70fcd5310f7e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:54.765551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2654280-9f43-40e9-afcc-0f34ca6c6af7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:54.766241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2654280-9f43-40e9-afcc-0f34ca6c6af7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:54.766525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2654280-9f43-40e9-afcc-0f34ca6c6af7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:54.766855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2654280-9f43-40e9-afcc-0f34ca6c6af7 None admin] PATH_INFO: `/v3/users/e481e627a19f4fcaa9af5c5635ce4374` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:54.768074 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2654280-9f43-40e9-afcc-0f34ca6c6af7 None admin] RBAC: Authorizing `identity:get_user(user_id=e481e627a19f4fcaa9af5c5635ce4374)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:54.769637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2654280-9f43-40e9-afcc-0f34ca6c6af7 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:54.771161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 68/132] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:00:54 2022] GET /identity/v3/users/e481e627a19f4fcaa9af5c5635ce4374 => generated 275 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:54.778873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2654280-9f43-40e9-afcc-0f34ca6c6af7 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:54.850285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15e790a-c786-4d51-b94e-2d0ee83a495a None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:54.897324 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2654280-9f43-40e9-afcc-0f34ca6c6af7 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:54.940906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd577fbd-a429-40fa-a479-dd4e3db97be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:54.941701 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd577fbd-a429-40fa-a479-dd4e3db97be2 None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:54.941986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd577fbd-a429-40fa-a479-dd4e3db97be2 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:54.942312 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd577fbd-a429-40fa-a479-dd4e3db97be2 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b/users/f82a94e2366b45ff82ac58061224a3a2/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:54.945001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd577fbd-a429-40fa-a479-dd4e3db97be2 None admin] RBAC: Authorizing `identity:create_grant(project_id=47fa4785636148129d33fffc2096cd4b, user_id=f82a94e2366b45ff82ac58061224a3a2, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:54.946571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd577fbd-a429-40fa-a479-dd4e3db97be2 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:54.949013 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2368773-6be2-4a8a-b008-3509e333d633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:54.949666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2368773-6be2-4a8a-b008-3509e333d633 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:54.949953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2368773-6be2-4a8a-b008-3509e333d633 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:54.950232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2368773-6be2-4a8a-b008-3509e333d633 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:54.951407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2368773-6be2-4a8a-b008-3509e333d633 None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:54.952880 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2368773-6be2-4a8a-b008-3509e333d633 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:54.954902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 65/133] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:00:54 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 366 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:54.955843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2368773-6be2-4a8a-b008-3509e333d633 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:54.966870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 69/134] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:00:54 2022] PUT /identity/v3/projects/47fa4785636148129d33fffc2096cd4b/users/f82a94e2366b45ff82ac58061224a3a2/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:00:54.972123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd577fbd-a429-40fa-a479-dd4e3db97be2 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:54.980928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:55.017329 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:55.176529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2368773-6be2-4a8a-b008-3509e333d633 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:55.186045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd577fbd-a429-40fa-a479-dd4e3db97be2 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:55.235030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec59cfb-0c12-4511-85d4-b31dc0f97725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:55.236196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec59cfb-0c12-4511-85d4-b31dc0f97725 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:55.236701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec59cfb-0c12-4511-85d4-b31dc0f97725 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:55.237289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec59cfb-0c12-4511-85d4-b31dc0f97725 None admin] PATH_INFO: `/v3/projects/b8dbeed180024b75b872ce3151dcc7f0` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:55.238835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec59cfb-0c12-4511-85d4-b31dc0f97725 None admin] RBAC: Authorizing `identity:get_project(project_id=b8dbeed180024b75b872ce3151dcc7f0)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:55.240250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483d203b-1488-43e9-8ea3-e06d6acdcc73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:55.241363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec59cfb-0c12-4511-85d4-b31dc0f97725 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:55.241467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483d203b-1488-43e9-8ea3-e06d6acdcc73 None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:55.241745 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483d203b-1488-43e9-8ea3-e06d6acdcc73 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:55.242019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483d203b-1488-43e9-8ea3-e06d6acdcc73 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b/users/e481e627a19f4fcaa9af5c5635ce4374/roles/a3902a185ac144fb8e24f4e9feb039ed` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:55.244032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 66/135] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:00:54 2022] GET /identity/v3/projects/b8dbeed180024b75b872ce3151dcc7f0 => generated 312 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:55.246743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483d203b-1488-43e9-8ea3-e06d6acdcc73 None admin] RBAC: Authorizing `identity:create_grant(project_id=47fa4785636148129d33fffc2096cd4b, user_id=e481e627a19f4fcaa9af5c5635ce4374, role_id=a3902a185ac144fb8e24f4e9feb039ed)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:55.246743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483d203b-1488-43e9-8ea3-e06d6acdcc73 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:55.248412 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362bf132-3840-42c9-982d-10974ca1be58 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:55.248933 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362bf132-3840-42c9-982d-10974ca1be58 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:55.249507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362bf132-3840-42c9-982d-10974ca1be58 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:55.251076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 67/136] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:55.255209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362bf132-3840-42c9-982d-10974ca1be58 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:55.264848 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 70/137] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:00:54 2022] PUT /identity/v3/projects/47fa4785636148129d33fffc2096cd4b/users/e481e627a19f4fcaa9af5c5635ce4374/roles/a3902a185ac144fb8e24f4e9feb039ed => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:00:55.266178 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab2512-0ce2-4d03-abca-b4b233b71184 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:55.266470 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab2512-0ce2-4d03-abca-b4b233b71184 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:55.266746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab2512-0ce2-4d03-abca-b4b233b71184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:55.291202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-63ab2512-0ce2-4d03-abca-b4b233b71184 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:55.292424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-63ab2512-0ce2-4d03-abca-b4b233b71184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:55.367542 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:55.383477 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362bf132-3840-42c9-982d-10974ca1be58 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:55.387315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:55.410679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 71/138] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:55.421879 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1649a27-2534-4c60-b5fc-b7fa557ba425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:55.422878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1649a27-2534-4c60-b5fc-b7fa557ba425 None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:55.423382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1649a27-2534-4c60-b5fc-b7fa557ba425 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:55.423855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1649a27-2534-4c60-b5fc-b7fa557ba425 None admin] PATH_INFO: `/v3/projects/b8dbeed180024b75b872ce3151dcc7f0/users/e481e627a19f4fcaa9af5c5635ce4374/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:55.426199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1649a27-2534-4c60-b5fc-b7fa557ba425 None admin] RBAC: Authorizing `identity:create_grant(project_id=b8dbeed180024b75b872ce3151dcc7f0, user_id=e481e627a19f4fcaa9af5c5635ce4374, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:55.427966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1649a27-2534-4c60-b5fc-b7fa557ba425 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:55.446256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 68/139] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:00:55 2022] PUT /identity/v3/projects/b8dbeed180024b75b872ce3151dcc7f0/users/e481e627a19f4fcaa9af5c5635ce4374/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:00:55.816704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ab2512-0ce2-4d03-abca-b4b233b71184 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:55.935081 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ab2512-0ce2-4d03-abca-b4b233b71184 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:55.983858 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf110f35-a1f9-49b0-9a56-ace872a15069 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:55.984528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf110f35-a1f9-49b0-9a56-ace872a15069 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:55.984815 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf110f35-a1f9-49b0-9a56-ace872a15069 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:55.985094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf110f35-a1f9-49b0-9a56-ace872a15069 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:55.986242 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf110f35-a1f9-49b0-9a56-ace872a15069 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:55.987720 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf110f35-a1f9-49b0-9a56-ace872a15069 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:55.989325 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 72/140] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:00:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:55.995575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1649a27-2534-4c60-b5fc-b7fa557ba425 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:56.156096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1649a27-2534-4c60-b5fc-b7fa557ba425 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:56.196445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2acb3fe-5940-4b38-be79-fd7b80530b3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:56.197056 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2acb3fe-5940-4b38-be79-fd7b80530b3f None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:56.197335 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2acb3fe-5940-4b38-be79-fd7b80530b3f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:56.197604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2acb3fe-5940-4b38-be79-fd7b80530b3f None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:56.198302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2acb3fe-5940-4b38-be79-fd7b80530b3f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:56.199756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2acb3fe-5940-4b38-be79-fd7b80530b3f None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:56.233156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 69/141] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:00:55 2022] POST /identity/v3/users => generated 283 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:00:56.495726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37ed114-51aa-4dd6-914f-d595caf2d3cc None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:56.496476 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37ed114-51aa-4dd6-914f-d595caf2d3cc None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:56.496908 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37ed114-51aa-4dd6-914f-d595caf2d3cc None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:56.498379 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 73/142] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:56 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:56.504017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d943c7-c035-4ff4-bbf8-51e816b213a3 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:56.504876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d943c7-c035-4ff4-bbf8-51e816b213a3 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:56.505297 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d943c7-c035-4ff4-bbf8-51e816b213a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:56.529489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-c9d943c7-c035-4ff4-bbf8-51e816b213a3 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:56.530981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c9d943c7-c035-4ff4-bbf8-51e816b213a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:56.533400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:56.622030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 70/143] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:56.942986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1e5b8a-da75-471d-810d-d13ee8fa2e76 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:56.943494 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1e5b8a-da75-471d-810d-d13ee8fa2e76 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:56.944487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1e5b8a-da75-471d-810d-d13ee8fa2e76 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:56.945999 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 74/144] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:56.951275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853136f1-b86f-421e-b4f4-f8298433d2a7 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:56.951572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853136f1-b86f-421e-b4f4-f8298433d2a7 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:56.951853 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853136f1-b86f-421e-b4f4-f8298433d2a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:56.981603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-853136f1-b86f-421e-b4f4-f8298433d2a7 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:56.982804 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-853136f1-b86f-421e-b4f4-f8298433d2a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:57.074565 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 71/145] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:57.103976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1e5b8a-da75-471d-810d-d13ee8fa2e76 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:57.190915 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:57.239702 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1e5b8a-da75-471d-810d-d13ee8fa2e76 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:57.277150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1ee2c4-13b6-4aaf-a380-3d09d22fe637 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:57.277829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1ee2c4-13b6-4aaf-a380-3d09d22fe637 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:57.278106 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1ee2c4-13b6-4aaf-a380-3d09d22fe637 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:57.278579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1ee2c4-13b6-4aaf-a380-3d09d22fe637 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:57.289514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1ee2c4-13b6-4aaf-a380-3d09d22fe637 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-de1ee2c4-13b6-4aaf-a380-3d09d22fe637 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:57.294455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:57.295275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:57.296046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:57.296046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:57.296046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:57.296046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:57.296046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:57.296046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:57.296046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:57.296046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:57.296046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:57.296046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:57.296046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:00:57.296046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:00:57.296046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 75/146] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:00:57 2022] GET /identity/v3/roles/reader => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:57.302659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853136f1-b86f-421e-b4f4-f8298433d2a7 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:57.384691 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:57.408857 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853136f1-b86f-421e-b4f4-f8298433d2a7 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:57.444196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b3148e-831e-43fb-9f06-847af541cfc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:57.445111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b3148e-831e-43fb-9f06-847af541cfc8 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:57.445561 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b3148e-831e-43fb-9f06-847af541cfc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:57.446002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b3148e-831e-43fb-9f06-847af541cfc8 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:57.446867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b3148e-831e-43fb-9f06-847af541cfc8 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:57.447383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b3148e-831e-43fb-9f06-847af541cfc8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:57.448953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b3148e-831e-43fb-9f06-847af541cfc8 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:57.457957 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 72/147] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:00:57 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:57.464090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1ee2c4-13b6-4aaf-a380-3d09d22fe637 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:57.551300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b3148e-831e-43fb-9f06-847af541cfc8 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:57.619949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1ee2c4-13b6-4aaf-a380-3d09d22fe637 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:57.643540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:57.677040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3325eae-9642-4e88-bb37-81c9fe8ba988 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:57.677694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3325eae-9642-4e88-bb37-81c9fe8ba988 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:57.678027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3325eae-9642-4e88-bb37-81c9fe8ba988 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:57.678311 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3325eae-9642-4e88-bb37-81c9fe8ba988 None admin] PATH_INFO: `/v3/users/dfc58f7c89394e26a4110f493c6fa02d` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:57.679536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3325eae-9642-4e88-bb37-81c9fe8ba988 None admin] RBAC: Authorizing `identity:get_user(user_id=dfc58f7c89394e26a4110f493c6fa02d)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:57.681026 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3325eae-9642-4e88-bb37-81c9fe8ba988 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:57.682611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 76/148] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:00:57 2022] GET /identity/v3/users/dfc58f7c89394e26a4110f493c6fa02d => generated 289 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:57.688128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3325eae-9642-4e88-bb37-81c9fe8ba988 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:57.790531 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b3148e-831e-43fb-9f06-847af541cfc8 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:57.874061 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:57.894532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30351aea-0de1-4c29-8082-9224b8233915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:57.895129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30351aea-0de1-4c29-8082-9224b8233915 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:57.895411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30351aea-0de1-4c29-8082-9224b8233915 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:57.895687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30351aea-0de1-4c29-8082-9224b8233915 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:57.906432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3325eae-9642-4e88-bb37-81c9fe8ba988 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:57.929368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30351aea-0de1-4c29-8082-9224b8233915 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-30351aea-0de1-4c29-8082-9224b8233915 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:57.934788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:57.935619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:57.936524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:57.936524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:57.936524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:57.936524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:57.936524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:57.936524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:57.936524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:57.936524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:57.936524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:57.936524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:57.936524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:00:57.936524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:00:57.936524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 73/149] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:00:57 2022] GET /identity/v3/roles/member => generated 84 bytes in 385 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:57.949368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30351aea-0de1-4c29-8082-9224b8233915 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:57.971708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480be8f9-ee64-4286-9e4d-d43901a1428f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:57.972727 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480be8f9-ee64-4286-9e4d-d43901a1428f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:57.973209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480be8f9-ee64-4286-9e4d-d43901a1428f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:57.973722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480be8f9-ee64-4286-9e4d-d43901a1428f None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:57.978010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480be8f9-ee64-4286-9e4d-d43901a1428f None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:57.979796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480be8f9-ee64-4286-9e4d-d43901a1428f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:57.982288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 77/150] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:00:57 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:00:57.987587 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480be8f9-ee64-4286-9e4d-d43901a1428f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:58.111257 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30351aea-0de1-4c29-8082-9224b8233915 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:58.133532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:58.159001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f86762-7778-48f2-a645-dc58117e2d6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:58.159628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f86762-7778-48f2-a645-dc58117e2d6e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:58.159909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f86762-7778-48f2-a645-dc58117e2d6e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:58.160183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f86762-7778-48f2-a645-dc58117e2d6e None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:58.160898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f86762-7778-48f2-a645-dc58117e2d6e None admin] RBAC: Adding query filter params (name=member) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:58.161203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f86762-7778-48f2-a645-dc58117e2d6e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:58.163206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f86762-7778-48f2-a645-dc58117e2d6e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:58.177605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 74/151] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:00:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:58.182938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f86762-7778-48f2-a645-dc58117e2d6e None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:58.250552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480be8f9-ee64-4286-9e4d-d43901a1428f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:58.257309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:58.326276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f86762-7778-48f2-a645-dc58117e2d6e None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:58.341058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f35b5e-3b3e-46fb-bb41-c1b9db35c3cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:58.341743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f35b5e-3b3e-46fb-bb41-c1b9db35c3cf None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:58.342020 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f35b5e-3b3e-46fb-bb41-c1b9db35c3cf None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:58.342294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f35b5e-3b3e-46fb-bb41-c1b9db35c3cf None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b/users/dfc58f7c89394e26a4110f493c6fa02d/roles/46cb171417764f9f9710877b8ec7becc` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:58.351614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f35b5e-3b3e-46fb-bb41-c1b9db35c3cf None admin] RBAC: Authorizing `identity:create_grant(project_id=47fa4785636148129d33fffc2096cd4b, user_id=dfc58f7c89394e26a4110f493c6fa02d, role_id=46cb171417764f9f9710877b8ec7becc)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:58.353131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f35b5e-3b3e-46fb-bb41-c1b9db35c3cf None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:58.370468 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfe9092-9ce9-4fc6-96be-4c22e1c70f68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:58.371029 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfe9092-9ce9-4fc6-96be-4c22e1c70f68 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:58.371313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfe9092-9ce9-4fc6-96be-4c22e1c70f68 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:58.371589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfe9092-9ce9-4fc6-96be-4c22e1c70f68 None admin] PATH_INFO: `/v3/users/e481e627a19f4fcaa9af5c5635ce4374` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:58.377071 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfe9092-9ce9-4fc6-96be-4c22e1c70f68 None admin] RBAC: Authorizing `identity:get_user(user_id=e481e627a19f4fcaa9af5c5635ce4374)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:58.378559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfe9092-9ce9-4fc6-96be-4c22e1c70f68 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:58.380373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 75/152] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:00:58 2022] GET /identity/v3/users/e481e627a19f4fcaa9af5c5635ce4374 => generated 275 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:00:58.382619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 78/153] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:00:57 2022] PUT /identity/v3/projects/47fa4785636148129d33fffc2096cd4b/users/dfc58f7c89394e26a4110f493c6fa02d/roles/46cb171417764f9f9710877b8ec7becc => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:00:58.392910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfe9092-9ce9-4fc6-96be-4c22e1c70f68 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:58.516258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfe9092-9ce9-4fc6-96be-4c22e1c70f68 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:58.544833 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:58.568928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0418a1-9b1b-46f2-ac01-ef4a50650771 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:58.569550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0418a1-9b1b-46f2-ac01-ef4a50650771 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:58.569823 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0418a1-9b1b-46f2-ac01-ef4a50650771 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:58.570097 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0418a1-9b1b-46f2-ac01-ef4a50650771 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:58.670333 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0418a1-9b1b-46f2-ac01-ef4a50650771 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:58.671759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0418a1-9b1b-46f2-ac01-ef4a50650771 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:58.735022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a1589f-7a23-43e8-92ee-e3a943138689 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:58.735528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a1589f-7a23-43e8-92ee-e3a943138689 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:58.735600 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a1589f-7a23-43e8-92ee-e3a943138689 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:58.737185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 79/154] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:58.752854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecfb549-0fc9-4af8-980c-67619c6d5455 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:58.753163 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecfb549-0fc9-4af8-980c-67619c6d5455 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:58.753432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecfb549-0fc9-4af8-980c-67619c6d5455 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:58.793467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-9ecfb549-0fc9-4af8-980c-67619c6d5455 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:58.794744 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9ecfb549-0fc9-4af8-980c-67619c6d5455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:58.817623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 76/155] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:00:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:00:58.827757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0418a1-9b1b-46f2-ac01-ef4a50650771 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:58.866478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:58.918823 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 80/156] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:58.978453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0418a1-9b1b-46f2-ac01-ef4a50650771 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:59.019301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b5495f-4c2a-49bf-9bc9-0f5f8ed08b32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:59.019935 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b5495f-4c2a-49bf-9bc9-0f5f8ed08b32 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:59.020214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b5495f-4c2a-49bf-9bc9-0f5f8ed08b32 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:59.020517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b5495f-4c2a-49bf-9bc9-0f5f8ed08b32 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:59.021770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b5495f-4c2a-49bf-9bc9-0f5f8ed08b32 None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:59.023343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b5495f-4c2a-49bf-9bc9-0f5f8ed08b32 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:59.025432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 77/157] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:00:58 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:59.031325 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecfb549-0fc9-4af8-980c-67619c6d5455 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:59.074401 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2023c9-c5bd-4520-b8c9-7126fdf640bc None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:59.074851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2023c9-c5bd-4520-b8c9-7126fdf640bc None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:59.075315 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2023c9-c5bd-4520-b8c9-7126fdf640bc None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:59.076921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 78/158] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:00:59.082667 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abbf825-585e-49cd-bc54-67b9d3648a9d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:59.083069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abbf825-585e-49cd-bc54-67b9d3648a9d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:59.083450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abbf825-585e-49cd-bc54-67b9d3648a9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:59.114834 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:59.117693 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-2abbf825-585e-49cd-bc54-67b9d3648a9d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:59.119137 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2abbf825-585e-49cd-bc54-67b9d3648a9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:59.210288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecfb549-0fc9-4af8-980c-67619c6d5455 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:59.222920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 79/159] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:59.224522 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaa7551-0c58-412d-a177-adf539085fc7 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:59.225694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaa7551-0c58-412d-a177-adf539085fc7 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:59.226172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaa7551-0c58-412d-a177-adf539085fc7 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:59.228186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 80/160] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:00:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:00:59.250583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba793aa0-8376-4f19-906b-af79e9d92af4 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:59.251183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba793aa0-8376-4f19-906b-af79e9d92af4 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:59.251667 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba793aa0-8376-4f19-906b-af79e9d92af4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:59.274204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca05566-fd5f-48e2-86c3-b4d4c8b4beef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:59.274875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca05566-fd5f-48e2-86c3-b4d4c8b4beef None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:59.275170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca05566-fd5f-48e2-86c3-b4d4c8b4beef None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:59.275445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca05566-fd5f-48e2-86c3-b4d4c8b4beef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:59.276193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca05566-fd5f-48e2-86c3-b4d4c8b4beef None admin] RBAC: Adding query filter params (role.id=615e83fab2504b6298db3efaa7c15750, scope.project.id=47fa4785636148129d33fffc2096cd4b, user.id=e481e627a19f4fcaa9af5c5635ce4374) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:00:59.276545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca05566-fd5f-48e2-86c3-b4d4c8b4beef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:00:59.278028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca05566-fd5f-48e2-86c3-b4d4c8b4beef None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:00:59.285403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-ba793aa0-8376-4f19-906b-af79e9d92af4 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:00:59.288608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ba793aa0-8376-4f19-906b-af79e9d92af4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:00:59.290233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 81/161] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:00:59 2022] GET /identity/v3/role_assignments?user.id=e481e627a19f4fcaa9af5c5635ce4374&scope.project.id=47fa4785636148129d33fffc2096cd4b&role.id=615e83fab2504b6298db3efaa7c15750 => generated 608 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:00:59.306846 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:59.377656 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 81/162] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:00:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:00:59.698857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca05566-fd5f-48e2-86c3-b4d4c8b4beef None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:59.805240 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca05566-fd5f-48e2-86c3-b4d4c8b4beef None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:59.816509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba793aa0-8376-4f19-906b-af79e9d92af4 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:59.817871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:00:59.848219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0685119e-0f72-4196-b9cb-3968dcea0bb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:59.848892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0685119e-0f72-4196-b9cb-3968dcea0bb7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:59.849173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0685119e-0f72-4196-b9cb-3968dcea0bb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:59.849448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0685119e-0f72-4196-b9cb-3968dcea0bb7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:00:59.871203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0685119e-0f72-4196-b9cb-3968dcea0bb7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-0685119e-0f72-4196-b9cb-3968dcea0bb7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:00:59.877044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:00:59.877936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:00:59.878728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:00:59.878728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:00:59.878728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:00:59.878728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:00:59.878728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:00:59.878728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:00:59.878728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:00:59.878728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:00:59.878728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:00:59.878728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:00:59.878728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:00:59.878728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:00:59.878728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 82/163] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:00:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:00:59.879453 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0685119e-0f72-4196-b9cb-3968dcea0bb7 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:00:59.931446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba793aa0-8376-4f19-906b-af79e9d92af4 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:00:59.979865 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674a587f-fba7-4df0-bf76-167a3c5ec691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:00:59.980453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a587f-fba7-4df0-bf76-167a3c5ec691 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:00:59.980732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a587f-fba7-4df0-bf76-167a3c5ec691 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:00:59.981002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a587f-fba7-4df0-bf76-167a3c5ec691 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:00.007563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674a587f-fba7-4df0-bf76-167a3c5ec691 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-674a587f-fba7-4df0-bf76-167a3c5ec691 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:00.021973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:00.022916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:00.023668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:00.023668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:00.023668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:00.023668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:00.023668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:00.023668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:00.023668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:00.023668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:00.023668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:00.023668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:00.023668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:01:00.023668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:00.028431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 82/164] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:00:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:00.029282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674a587f-fba7-4df0-bf76-167a3c5ec691 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:00.056681 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:00.090384 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0685119e-0f72-4196-b9cb-3968dcea0bb7 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:00.148668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee0d305-6ca2-4233-99fa-b75468f58740 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:00.149312 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee0d305-6ca2-4233-99fa-b75468f58740 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:00.149589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee0d305-6ca2-4233-99fa-b75468f58740 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:00.149864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee0d305-6ca2-4233-99fa-b75468f58740 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:00.173115 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee0d305-6ca2-4233-99fa-b75468f58740 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-2ee0d305-6ca2-4233-99fa-b75468f58740 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:00.182186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:00.183057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:00.183836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:00.183836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:00.183836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:00.183836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:00.183836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:00.183836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:00.183836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:00.183836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:00.183836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:00.183836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:00.183836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:00.183836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:00.183836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 83/165] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:00:59 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:00.184539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee0d305-6ca2-4233-99fa-b75468f58740 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:00.218130 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674a587f-fba7-4df0-bf76-167a3c5ec691 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:00.252632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:00.265987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe46845-2f41-4726-a7fb-dbb802e23838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:00.266611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe46845-2f41-4726-a7fb-dbb802e23838 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:00.266892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe46845-2f41-4726-a7fb-dbb802e23838 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:00.267163 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe46845-2f41-4726-a7fb-dbb802e23838 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:00.267847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe46845-2f41-4726-a7fb-dbb802e23838 None admin] RBAC: Adding query filter params (name=member) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:00.268153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe46845-2f41-4726-a7fb-dbb802e23838 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:00.269559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe46845-2f41-4726-a7fb-dbb802e23838 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:00.283879 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 83/166] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:00.285816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe46845-2f41-4726-a7fb-dbb802e23838 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:00.418193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee0d305-6ca2-4233-99fa-b75468f58740 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:00.487794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f7e2cd-fb78-4daf-bb1a-e96dc14e76b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:00.488819 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f7e2cd-fb78-4daf-bb1a-e96dc14e76b1 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:00.489325 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f7e2cd-fb78-4daf-bb1a-e96dc14e76b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:00.489819 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f7e2cd-fb78-4daf-bb1a-e96dc14e76b1 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:00.491230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f7e2cd-fb78-4daf-bb1a-e96dc14e76b1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:00.491744 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f7e2cd-fb78-4daf-bb1a-e96dc14e76b1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:00.493380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f7e2cd-fb78-4daf-bb1a-e96dc14e76b1 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:00.506138 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:00.511596 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 84/167] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:00.512492 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f7e2cd-fb78-4daf-bb1a-e96dc14e76b1 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:00.598203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe46845-2f41-4726-a7fb-dbb802e23838 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:00.692090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:00.701022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2505572-91ea-4d14-b2e1-279f8a9c625b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:00.701644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2505572-91ea-4d14-b2e1-279f8a9c625b None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:00.701925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2505572-91ea-4d14-b2e1-279f8a9c625b None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:00.702199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2505572-91ea-4d14-b2e1-279f8a9c625b None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:00.702881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2505572-91ea-4d14-b2e1-279f8a9c625b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:00.703184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2505572-91ea-4d14-b2e1-279f8a9c625b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:00.704594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2505572-91ea-4d14-b2e1-279f8a9c625b None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:00.707816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f7e2cd-fb78-4daf-bb1a-e96dc14e76b1 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:00.723312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 84/168] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:00 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 439 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:00.724233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2505572-91ea-4d14-b2e1-279f8a9c625b None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:00.756836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0255f1d-d25d-4ca3-b6f3-f25079adc4b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:00.757894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0255f1d-d25d-4ca3-b6f3-f25079adc4b6 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:00.758184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0255f1d-d25d-4ca3-b6f3-f25079adc4b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:00.758618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0255f1d-d25d-4ca3-b6f3-f25079adc4b6 None admin] PATH_INFO: `/v3/users/e481e627a19f4fcaa9af5c5635ce4374` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:00.759789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0255f1d-d25d-4ca3-b6f3-f25079adc4b6 None admin] RBAC: Authorizing `identity:get_user(user_id=e481e627a19f4fcaa9af5c5635ce4374)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:00.761473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0255f1d-d25d-4ca3-b6f3-f25079adc4b6 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:00.763536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 85/169] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:00 2022] GET /identity/v3/users/e481e627a19f4fcaa9af5c5635ce4374 => generated 275 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:00.764397 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0255f1d-d25d-4ca3-b6f3-f25079adc4b6 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:00.926513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2505572-91ea-4d14-b2e1-279f8a9c625b None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:00.943077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0255f1d-d25d-4ca3-b6f3-f25079adc4b6 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:00.956699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:00.974842 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1867cd4c-90a1-4b3e-88ce-7e80f29c5e78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:00.975493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1867cd4c-90a1-4b3e-88ce-7e80f29c5e78 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:00.975772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1867cd4c-90a1-4b3e-88ce-7e80f29c5e78 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:00.976043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1867cd4c-90a1-4b3e-88ce-7e80f29c5e78 None admin] PATH_INFO: `/v3/users/f82a94e2366b45ff82ac58061224a3a2` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:00.977960 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1867cd4c-90a1-4b3e-88ce-7e80f29c5e78 None admin] RBAC: Authorizing `identity:get_user(user_id=f82a94e2366b45ff82ac58061224a3a2)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:00.979399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1867cd4c-90a1-4b3e-88ce-7e80f29c5e78 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:00.981154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 85/170] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:00 2022] GET /identity/v3/users/f82a94e2366b45ff82ac58061224a3a2 => generated 247 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:00.982961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1867cd4c-90a1-4b3e-88ce-7e80f29c5e78 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:01.026675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d7d816-6e55-49e1-b3c3-00f73ab39a20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:01.027771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d7d816-6e55-49e1-b3c3-00f73ab39a20 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:01.028283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d7d816-6e55-49e1-b3c3-00f73ab39a20 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:01.028896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d7d816-6e55-49e1-b3c3-00f73ab39a20 None admin] PATH_INFO: `/v3/users/e481e627a19f4fcaa9af5c5635ce4374` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:01.030652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d7d816-6e55-49e1-b3c3-00f73ab39a20 None admin] RBAC: Authorizing `identity:get_user(user_id=e481e627a19f4fcaa9af5c5635ce4374)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:01.032834 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d7d816-6e55-49e1-b3c3-00f73ab39a20 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:01.035472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 86/171] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:00 2022] GET /identity/v3/users/e481e627a19f4fcaa9af5c5635ce4374 => generated 275 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:01.038578 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e270d4-cdeb-4b32-8ba6-291152b40d06 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:01.039846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e270d4-cdeb-4b32-8ba6-291152b40d06 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:01.041058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e270d4-cdeb-4b32-8ba6-291152b40d06 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:01.048559 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 87/172] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:01.048559 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e270d4-cdeb-4b32-8ba6-291152b40d06 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:01.104857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:01.195770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1867cd4c-90a1-4b3e-88ce-7e80f29c5e78 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:01.211091 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e270d4-cdeb-4b32-8ba6-291152b40d06 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:01.263020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be264a03-5000-4696-8e7d-f0c34535b8ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:01.264002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be264a03-5000-4696-8e7d-f0c34535b8ea None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:01.264521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be264a03-5000-4696-8e7d-f0c34535b8ea None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:01.265054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be264a03-5000-4696-8e7d-f0c34535b8ea None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:01.272240 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58e3f90-4b59-4dd4-a082-c8ac64ec90bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:01.274529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58e3f90-4b59-4dd4-a082-c8ac64ec90bc None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:01.275011 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58e3f90-4b59-4dd4-a082-c8ac64ec90bc None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:01.277115 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58e3f90-4b59-4dd4-a082-c8ac64ec90bc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:01.390065 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:01.395046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be264a03-5000-4696-8e7d-f0c34535b8ea None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:01.396587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be264a03-5000-4696-8e7d-f0c34535b8ea None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:01.408045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58e3f90-4b59-4dd4-a082-c8ac64ec90bc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:01.409578 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58e3f90-4b59-4dd4-a082-c8ac64ec90bc None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:01.449631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:01.523012 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 86/173] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 541 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:01.524237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be264a03-5000-4696-8e7d-f0c34535b8ea None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:01.603482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 88/174] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 560 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:01.608359 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1bb1d1-402e-4df1-8531-233dd52148b7 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:01.608359 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1bb1d1-402e-4df1-8531-233dd52148b7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:01.608359 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1bb1d1-402e-4df1-8531-233dd52148b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:01.627107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-af1bb1d1-402e-4df1-8531-233dd52148b7 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:01.629053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-af1bb1d1-402e-4df1-8531-233dd52148b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:01.648237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:01.654394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be264a03-5000-4696-8e7d-f0c34535b8ea None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:01.697423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8787b92-3eb4-4735-a41f-51e0bd211b05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:01.697773 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8787b92-3eb4-4735-a41f-51e0bd211b05 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:01.697940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8787b92-3eb4-4735-a41f-51e0bd211b05 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:01.698218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8787b92-3eb4-4735-a41f-51e0bd211b05 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:01.713468 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 89/175] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:01.714840 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1bb1d1-402e-4df1-8531-233dd52148b7 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:01.780101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:01.782320 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8787b92-3eb4-4735-a41f-51e0bd211b05 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:01.783802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8787b92-3eb4-4735-a41f-51e0bd211b05 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:01.841326 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1bb1d1-402e-4df1-8531-233dd52148b7 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:01.914723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ae25a9-31bd-4e33-8d2c-98c9863e6347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:01.915287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ae25a9-31bd-4e33-8d2c-98c9863e6347 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:01.915568 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ae25a9-31bd-4e33-8d2c-98c9863e6347 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:01.915840 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ae25a9-31bd-4e33-8d2c-98c9863e6347 None admin] PATH_INFO: `/v3/projects/b8dbeed180024b75b872ce3151dcc7f0` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:01.920627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ae25a9-31bd-4e33-8d2c-98c9863e6347 None admin] RBAC: Authorizing `identity:get_project(project_id=b8dbeed180024b75b872ce3151dcc7f0)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:01.922118 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ae25a9-31bd-4e33-8d2c-98c9863e6347 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:01.923835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 90/176] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:01 2022] GET /identity/v3/projects/b8dbeed180024b75b872ce3151dcc7f0 => generated 312 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:01.933780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d281c3d-5a92-4e2c-ab54-467eef55db12 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:01.934062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d281c3d-5a92-4e2c-ab54-467eef55db12 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:01.934338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d281c3d-5a92-4e2c-ab54-467eef55db12 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:01.935647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 91/177] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:01 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:01.936371 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d281c3d-5a92-4e2c-ab54-467eef55db12 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:01.967145 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:01.971325 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 87/178] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 447 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:01.972677 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcf92d8-7245-45c6-b165-ce09244f80a7 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:01.973083 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcf92d8-7245-45c6-b165-ce09244f80a7 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:01.973363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcf92d8-7245-45c6-b165-ce09244f80a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:02.022560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-efcf92d8-7245-45c6-b165-ce09244f80a7 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:02.023889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-efcf92d8-7245-45c6-b165-ce09244f80a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:02.064058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d281c3d-5a92-4e2c-ab54-467eef55db12 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:02.097005 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:02.104588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 88/179] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:02.105500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcf92d8-7245-45c6-b165-ce09244f80a7 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:02.117670 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb3c755-d24d-43d1-841a-8a0319b56f7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:02.119102 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb3c755-d24d-43d1-841a-8a0319b56f7c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:02.119935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb3c755-d24d-43d1-841a-8a0319b56f7c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:02.120425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb3c755-d24d-43d1-841a-8a0319b56f7c None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:02.122113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb3c755-d24d-43d1-841a-8a0319b56f7c None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:02.124636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb3c755-d24d-43d1-841a-8a0319b56f7c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:02.127093 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 92/180] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:01 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:02.127806 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb3c755-d24d-43d1-841a-8a0319b56f7c None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:02.211553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcf92d8-7245-45c6-b165-ce09244f80a7 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:02.237951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb3c755-d24d-43d1-841a-8a0319b56f7c None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:02.242800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:02.250190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9957f95a-f8e4-4264-827d-041a352f3e82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:02.250794 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9957f95a-f8e4-4264-827d-041a352f3e82 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:02.251068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9957f95a-f8e4-4264-827d-041a352f3e82 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:02.251438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9957f95a-f8e4-4264-827d-041a352f3e82 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:02.252062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9957f95a-f8e4-4264-827d-041a352f3e82 None admin] RBAC: Adding query filter params (role.id=615e83fab2504b6298db3efaa7c15750, scope.project.id=b8dbeed180024b75b872ce3151dcc7f0, user.id=e481e627a19f4fcaa9af5c5635ce4374) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:02.252389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9957f95a-f8e4-4264-827d-041a352f3e82 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:02.253800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9957f95a-f8e4-4264-827d-041a352f3e82 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:02.259674 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 89/181] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:02 2022] GET /identity/v3/role_assignments?user.id=e481e627a19f4fcaa9af5c5635ce4374&scope.project.id=b8dbeed180024b75b872ce3151dcc7f0&role.id=615e83fab2504b6298db3efaa7c15750 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:02.268661 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9957f95a-f8e4-4264-827d-041a352f3e82 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:02.285710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239f8bc1-e8ca-4476-8bc9-d4abbb013329 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:02.286648 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239f8bc1-e8ca-4476-8bc9-d4abbb013329 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:02.287051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239f8bc1-e8ca-4476-8bc9-d4abbb013329 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:02.287940 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239f8bc1-e8ca-4476-8bc9-d4abbb013329 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:02.289349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239f8bc1-e8ca-4476-8bc9-d4abbb013329 None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:02.291025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239f8bc1-e8ca-4476-8bc9-d4abbb013329 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:02.293079 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 93/182] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:02 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:02.294485 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4223094d-a0d2-4aa5-b22b-ba130b836af8 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:02.294839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4223094d-a0d2-4aa5-b22b-ba130b836af8 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:02.295182 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4223094d-a0d2-4aa5-b22b-ba130b836af8 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:02.296794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 94/183] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:02.298058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81945e3-d2df-4d5e-b98e-80f012ddd29a None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:02.298176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81945e3-d2df-4d5e-b98e-80f012ddd29a None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:02.298482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81945e3-d2df-4d5e-b98e-80f012ddd29a None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:02.299808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 95/184] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:02.300644 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81945e3-d2df-4d5e-b98e-80f012ddd29a None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:02.385002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:02.400791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9957f95a-f8e4-4264-827d-041a352f3e82 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:02.418768 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81945e3-d2df-4d5e-b98e-80f012ddd29a None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:02.447288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707e5f64-6b20-4bd6-a0c0-68c4650eac11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:02.448102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707e5f64-6b20-4bd6-a0c0-68c4650eac11 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:02.448227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707e5f64-6b20-4bd6-a0c0-68c4650eac11 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:02.448508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707e5f64-6b20-4bd6-a0c0-68c4650eac11 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:02.460108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be0fe10-013e-43dd-9dd9-3311ce58caf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:02.460755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be0fe10-013e-43dd-9dd9-3311ce58caf1 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:02.461041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be0fe10-013e-43dd-9dd9-3311ce58caf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:02.461416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be0fe10-013e-43dd-9dd9-3311ce58caf1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:02.463128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be0fe10-013e-43dd-9dd9-3311ce58caf1 None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=47fa4785636148129d33fffc2096cd4b, user.id=f82a94e2366b45ff82ac58061224a3a2) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:02.463646 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be0fe10-013e-43dd-9dd9-3311ce58caf1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:02.465186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707e5f64-6b20-4bd6-a0c0-68c4650eac11 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:02.465500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be0fe10-013e-43dd-9dd9-3311ce58caf1 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-707e5f64-6b20-4bd6-a0c0-68c4650eac11 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:02.470523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:02.471384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:02.472156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:02.472156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:02.472156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:02.472156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:02.472156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:02.472156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:02.472156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:02.472156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:02.472156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:02.472156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:02.472156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:01:02.472156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:02.472156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 90/185] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:01:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:02.472873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707e5f64-6b20-4bd6-a0c0-68c4650eac11 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:02.480811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 96/186] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:02 2022] GET /identity/v3/role_assignments?user.id=f82a94e2366b45ff82ac58061224a3a2&scope.project.id=47fa4785636148129d33fffc2096cd4b&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:02.480811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b164712-e358-4d91-9d87-6f7a7e49796f None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:02.480811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b164712-e358-4d91-9d87-6f7a7e49796f None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:02.480811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b164712-e358-4d91-9d87-6f7a7e49796f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:02.500042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3b164712-e358-4d91-9d87-6f7a7e49796f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:02.500243 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3b164712-e358-4d91-9d87-6f7a7e49796f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:02.565875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:02.603300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 97/187] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:02.604678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4235ac1-54b3-4652-b87f-c73836ee4887 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:02.604968 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4235ac1-54b3-4652-b87f-c73836ee4887 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:02.605244 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4235ac1-54b3-4652-b87f-c73836ee4887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:02.624470 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-b4235ac1-54b3-4652-b87f-c73836ee4887 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:02.625584 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b4235ac1-54b3-4652-b87f-c73836ee4887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:02.632388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707e5f64-6b20-4bd6-a0c0-68c4650eac11 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:02.676190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fe888c-0c7f-4201-b240-bfec2205e652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:02.676979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fe888c-0c7f-4201-b240-bfec2205e652 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:02.677263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fe888c-0c7f-4201-b240-bfec2205e652 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:02.677670 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fe888c-0c7f-4201-b240-bfec2205e652 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:02.678435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31fe888c-0c7f-4201-b240-bfec2205e652 None admin] RBAC: Adding query filter params (role.id=a3902a185ac144fb8e24f4e9feb039ed, scope.project.id=47fa4785636148129d33fffc2096cd4b, user.id=e481e627a19f4fcaa9af5c5635ce4374) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:02.678938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31fe888c-0c7f-4201-b240-bfec2205e652 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:02.681125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31fe888c-0c7f-4201-b240-bfec2205e652 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:02.683085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:02.688693 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 91/188] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:02 2022] GET /identity/v3/role_assignments?user.id=e481e627a19f4fcaa9af5c5635ce4374&scope.project.id=47fa4785636148129d33fffc2096cd4b&role.id=a3902a185ac144fb8e24f4e9feb039ed => generated 608 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:02.692843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fe888c-0c7f-4201-b240-bfec2205e652 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:02.720450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 98/189] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:02.721322 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4235ac1-54b3-4652-b87f-c73836ee4887 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:02.875093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fe888c-0c7f-4201-b240-bfec2205e652 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:02.897061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4235ac1-54b3-4652-b87f-c73836ee4887 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:02.909408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:02.935897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0e9157-a5c5-4389-8d5d-539a091f0eef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:02.936570 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0e9157-a5c5-4389-8d5d-539a091f0eef None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:02.936849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0e9157-a5c5-4389-8d5d-539a091f0eef None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:02.937121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0e9157-a5c5-4389-8d5d-539a091f0eef None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:02.951522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e8d1bf-1072-403f-9d97-e33de9c3e3bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:02.952137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e8d1bf-1072-403f-9d97-e33de9c3e3bc None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:02.952517 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e8d1bf-1072-403f-9d97-e33de9c3e3bc None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:02.952791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e8d1bf-1072-403f-9d97-e33de9c3e3bc None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:02.953465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e8d1bf-1072-403f-9d97-e33de9c3e3bc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:02.953765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e8d1bf-1072-403f-9d97-e33de9c3e3bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:02.955160 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e8d1bf-1072-403f-9d97-e33de9c3e3bc None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:02.957114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0e9157-a5c5-4389-8d5d-539a091f0eef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-4e0e9157-a5c5-4389-8d5d-539a091f0eef None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:02.963653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:02.964545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:02.965354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:02.965354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:02.965354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:02.965354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:02.965354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:02.965354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:02.965354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:02.965354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:02.965354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:02.965354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:02.965354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:02.965354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:02.966007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 99/190] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:02.966007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e8d1bf-1072-403f-9d97-e33de9c3e3bc None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:02.967825 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 92/191] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:01:02 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:02.970173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0e9157-a5c5-4389-8d5d-539a091f0eef None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:03.038466 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:03.090268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0e9157-a5c5-4389-8d5d-539a091f0eef None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:03.096992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e8d1bf-1072-403f-9d97-e33de9c3e3bc None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:03.129875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230a64de-fa9b-4d20-9a26-77620ddb3c06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:03.130622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230a64de-fa9b-4d20-9a26-77620ddb3c06 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:03.130800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230a64de-fa9b-4d20-9a26-77620ddb3c06 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:03.131623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230a64de-fa9b-4d20-9a26-77620ddb3c06 None admin] PATH_INFO: `/v3/users/1f696d5af6cc481c9ccbd6e4f054f0a1` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:03.138339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003402cf-3713-43d7-bbef-1fc8c1b621fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:03.138992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003402cf-3713-43d7-bbef-1fc8c1b621fe None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:03.139271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003402cf-3713-43d7-bbef-1fc8c1b621fe None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:03.139544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003402cf-3713-43d7-bbef-1fc8c1b621fe None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:03.145378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230a64de-fa9b-4d20-9a26-77620ddb3c06 None admin] RBAC: Authorizing `identity:get_user(user_id=1f696d5af6cc481c9ccbd6e4f054f0a1)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:03.146850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230a64de-fa9b-4d20-9a26-77620ddb3c06 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:03.148511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 93/192] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:02 2022] GET /identity/v3/users/1f696d5af6cc481c9ccbd6e4f054f0a1 => generated 283 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:03.149271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230a64de-fa9b-4d20-9a26-77620ddb3c06 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:03.150015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003402cf-3713-43d7-bbef-1fc8c1b621fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-003402cf-3713-43d7-bbef-1fc8c1b621fe None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:03.155331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:03.156250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:03.157099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:03.157099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:03.157099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:03.157099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:03.157099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:03.157099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:03.157099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:03.157099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:03.157099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:03.157099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:03.157099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:01:03.157099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:03.157099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 100/193] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:02 2022] GET /identity/v3/roles/reader => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:03.157766 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003402cf-3713-43d7-bbef-1fc8c1b621fe None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:03.189636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:03.252782 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230a64de-fa9b-4d20-9a26-77620ddb3c06 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:03.254349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003402cf-3713-43d7-bbef-1fc8c1b621fe None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:03.263253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:03.286611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91a3cba-639c-4e52-b89e-e4eb0695c8da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:03.287236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91a3cba-639c-4e52-b89e-e4eb0695c8da None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:03.287511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91a3cba-639c-4e52-b89e-e4eb0695c8da None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:03.287778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91a3cba-639c-4e52-b89e-e4eb0695c8da None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:03.287965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b3cab6-af5f-425a-b61c-0e5451b4878c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:03.288533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91a3cba-639c-4e52-b89e-e4eb0695c8da None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:03.288632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b3cab6-af5f-425a-b61c-0e5451b4878c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:03.288786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91a3cba-639c-4e52-b89e-e4eb0695c8da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:03.288876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b3cab6-af5f-425a-b61c-0e5451b4878c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:03.289149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b3cab6-af5f-425a-b61c-0e5451b4878c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:03.290140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91a3cba-639c-4e52-b89e-e4eb0695c8da None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:03.290270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b3cab6-af5f-425a-b61c-0e5451b4878c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:03.291724 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b3cab6-af5f-425a-b61c-0e5451b4878c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:03.293320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 101/194] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:03.294090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b3cab6-af5f-425a-b61c-0e5451b4878c None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:03.302545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 94/195] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:03.302545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91a3cba-639c-4e52-b89e-e4eb0695c8da None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:03.392657 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:03.393129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b3cab6-af5f-425a-b61c-0e5451b4878c None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:03.397886 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91a3cba-639c-4e52-b89e-e4eb0695c8da None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:03.426116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c0ed17-ae11-4fbb-9c2d-d33ac4c80eb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:03.426713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c0ed17-ae11-4fbb-9c2d-d33ac4c80eb7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:03.426993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c0ed17-ae11-4fbb-9c2d-d33ac4c80eb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:03.427267 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c0ed17-ae11-4fbb-9c2d-d33ac4c80eb7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:03.430778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c22adad-bf13-4d97-8628-437168f95be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:03.431370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c22adad-bf13-4d97-8628-437168f95be2 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:03.431644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c22adad-bf13-4d97-8628-437168f95be2 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:03.431910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c22adad-bf13-4d97-8628-437168f95be2 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:03.432604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c22adad-bf13-4d97-8628-437168f95be2 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:03.432903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c22adad-bf13-4d97-8628-437168f95be2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:03.434338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c22adad-bf13-4d97-8628-437168f95be2 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:03.444132 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 95/196] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:03 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:03.444976 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c22adad-bf13-4d97-8628-437168f95be2 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:03.487626 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:03.512706 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c0ed17-ae11-4fbb-9c2d-d33ac4c80eb7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:03.514238 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c0ed17-ae11-4fbb-9c2d-d33ac4c80eb7 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:03.549962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c22adad-bf13-4d97-8628-437168f95be2 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:03.581723 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd34008-585e-49cd-9a7e-7d1577f4c0a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:03.582318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd34008-585e-49cd-9a7e-7d1577f4c0a5 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:03.582593 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd34008-585e-49cd-9a7e-7d1577f4c0a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:03.582859 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd34008-585e-49cd-9a7e-7d1577f4c0a5 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:03.583499 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd34008-585e-49cd-9a7e-7d1577f4c0a5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:03.584986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd34008-585e-49cd-9a7e-7d1577f4c0a5 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:03.606860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 96/197] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:01:03 2022] POST /identity/v3/users => generated 297 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:01:03.609344 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd34008-585e-49cd-9a7e-7d1577f4c0a5 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:03.612375 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 102/198] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:03.613308 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c0ed17-ae11-4fbb-9c2d-d33ac4c80eb7 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:03.631722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:03.713308 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:03.729360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd34008-585e-49cd-9a7e-7d1577f4c0a5 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:03.740669 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c0ed17-ae11-4fbb-9c2d-d33ac4c80eb7 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:03.769303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4343e98e-180f-409f-89a0-18d37c672754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:03.771389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4343e98e-180f-409f-89a0-18d37c672754 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:03.772466 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4343e98e-180f-409f-89a0-18d37c672754 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:03.772754 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4343e98e-180f-409f-89a0-18d37c672754 None admin] PATH_INFO: `/v3/users/1f696d5af6cc481c9ccbd6e4f054f0a1` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:03.774241 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4343e98e-180f-409f-89a0-18d37c672754 None admin] RBAC: Authorizing `identity:get_user(user_id=1f696d5af6cc481c9ccbd6e4f054f0a1)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:03.776394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4343e98e-180f-409f-89a0-18d37c672754 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:03.778331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 97/199] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:03 2022] GET /identity/v3/users/1f696d5af6cc481c9ccbd6e4f054f0a1 => generated 283 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:03.779468 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4343e98e-180f-409f-89a0-18d37c672754 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:03.781422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d29cd8-e9ac-44b0-b9cb-61be4363e7ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:03.782228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d29cd8-e9ac-44b0-b9cb-61be4363e7ce None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:03.782648 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d29cd8-e9ac-44b0-b9cb-61be4363e7ce None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:03.783056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d29cd8-e9ac-44b0-b9cb-61be4363e7ce None admin] PATH_INFO: `/v3/users/dfc58f7c89394e26a4110f493c6fa02d` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:03.785758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d29cd8-e9ac-44b0-b9cb-61be4363e7ce None admin] RBAC: Authorizing `identity:get_user(user_id=dfc58f7c89394e26a4110f493c6fa02d)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:03.787238 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d29cd8-e9ac-44b0-b9cb-61be4363e7ce None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:03.789347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 103/200] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:03 2022] GET /identity/v3/users/dfc58f7c89394e26a4110f493c6fa02d => generated 289 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:03.793161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d29cd8-e9ac-44b0-b9cb-61be4363e7ce None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:03.881539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:03.903874 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4343e98e-180f-409f-89a0-18d37c672754 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:03.921415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d29cd8-e9ac-44b0-b9cb-61be4363e7ce None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:03.951721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60de5121-76d8-46a6-a7f7-da31af22ce1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:03.952514 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60de5121-76d8-46a6-a7f7-da31af22ce1e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:03.952860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60de5121-76d8-46a6-a7f7-da31af22ce1e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:03.953203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60de5121-76d8-46a6-a7f7-da31af22ce1e None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:03.954436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60de5121-76d8-46a6-a7f7-da31af22ce1e None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:03.955935 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60de5121-76d8-46a6-a7f7-da31af22ce1e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:03.957716 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 98/201] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:03 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:03.959573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60de5121-76d8-46a6-a7f7-da31af22ce1e None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:03.967838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2dd497-e88c-4d42-a6e2-1b7c641ce23f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:03.968764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2dd497-e88c-4d42-a6e2-1b7c641ce23f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:03.969048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2dd497-e88c-4d42-a6e2-1b7c641ce23f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:03.969320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2dd497-e88c-4d42-a6e2-1b7c641ce23f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:03.986915 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:04.058469 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2dd497-e88c-4d42-a6e2-1b7c641ce23f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:04.060357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2dd497-e88c-4d42-a6e2-1b7c641ce23f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:04.087989 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60de5121-76d8-46a6-a7f7-da31af22ce1e None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:04.135767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e7a639-8b78-462c-bfe8-a49a5612f9ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:04.136328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e7a639-8b78-462c-bfe8-a49a5612f9ba None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:04.136610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e7a639-8b78-462c-bfe8-a49a5612f9ba None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:04.136891 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e7a639-8b78-462c-bfe8-a49a5612f9ba None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:04.161736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:04.192672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 104/202] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:04.198037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2dd497-e88c-4d42-a6e2-1b7c641ce23f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:04.217235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e7a639-8b78-462c-bfe8-a49a5612f9ba None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:04.219105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e7a639-8b78-462c-bfe8-a49a5612f9ba None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:04.247993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:04.307219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2dd497-e88c-4d42-a6e2-1b7c641ce23f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:04.329573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 99/203] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:04.330442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e7a639-8b78-462c-bfe8-a49a5612f9ba None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:04.347179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7ff1f9-918d-49b9-a2f2-b4ffab79520d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:04.347785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7ff1f9-918d-49b9-a2f2-b4ffab79520d None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:04.348063 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7ff1f9-918d-49b9-a2f2-b4ffab79520d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:04.348358 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7ff1f9-918d-49b9-a2f2-b4ffab79520d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:04.349243 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7ff1f9-918d-49b9-a2f2-b4ffab79520d None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4, user.id=1f696d5af6cc481c9ccbd6e4f054f0a1) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:04.350094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7ff1f9-918d-49b9-a2f2-b4ffab79520d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:04.351537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7ff1f9-918d-49b9-a2f2-b4ffab79520d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:04.357542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 105/204] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:04 2022] GET /identity/v3/role_assignments?user.id=1f696d5af6cc481c9ccbd6e4f054f0a1&scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:04.358265 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7ff1f9-918d-49b9-a2f2-b4ffab79520d None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:04.413013 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:04.474395 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e7a639-8b78-462c-bfe8-a49a5612f9ba None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:04.515159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5895fefb-7bde-4f18-be3d-46b8c5796a01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:04.515800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5895fefb-7bde-4f18-be3d-46b8c5796a01 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:04.516081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5895fefb-7bde-4f18-be3d-46b8c5796a01 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:04.516403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5895fefb-7bde-4f18-be3d-46b8c5796a01 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:04.517885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5895fefb-7bde-4f18-be3d-46b8c5796a01 None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:04.519624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5895fefb-7bde-4f18-be3d-46b8c5796a01 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:04.522120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 100/205] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:04 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:04.528584 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5895fefb-7bde-4f18-be3d-46b8c5796a01 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:04.640090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7ff1f9-918d-49b9-a2f2-b4ffab79520d None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:04.650047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5895fefb-7bde-4f18-be3d-46b8c5796a01 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:04.655715 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:04.680766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:04.696890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b82305e-0979-4af9-a1e3-dee03e696920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:04.697507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b82305e-0979-4af9-a1e3-dee03e696920 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:04.697785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b82305e-0979-4af9-a1e3-dee03e696920 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:04.698064 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b82305e-0979-4af9-a1e3-dee03e696920 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:04.698800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b82305e-0979-4af9-a1e3-dee03e696920 None admin] RBAC: Adding query filter params (role.id=a3902a185ac144fb8e24f4e9feb039ed, scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4, user.id=1f696d5af6cc481c9ccbd6e4f054f0a1) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:04.699167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b82305e-0979-4af9-a1e3-dee03e696920 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:04.700633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b82305e-0979-4af9-a1e3-dee03e696920 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:04.712587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 101/206] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:04 2022] GET /identity/v3/role_assignments?user.id=1f696d5af6cc481c9ccbd6e4f054f0a1&scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4&role.id=a3902a185ac144fb8e24f4e9feb039ed => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:04.717101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c116c888-e675-4ce4-844e-b79f57c00825 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:04.718142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c116c888-e675-4ce4-844e-b79f57c00825 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:04.718643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c116c888-e675-4ce4-844e-b79f57c00825 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:04.719140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c116c888-e675-4ce4-844e-b79f57c00825 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:04.725367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c116c888-e675-4ce4-844e-b79f57c00825 None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:04.727317 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c116c888-e675-4ce4-844e-b79f57c00825 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:04.729399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 106/207] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:04 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 371 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:04.735623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b82305e-0979-4af9-a1e3-dee03e696920 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:04.859061 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b82305e-0979-4af9-a1e3-dee03e696920 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:04.895417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df908859-87a7-4d65-9a23-06ce741095f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:04.896032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df908859-87a7-4d65-9a23-06ce741095f7 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:04.896336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df908859-87a7-4d65-9a23-06ce741095f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:04.896652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df908859-87a7-4d65-9a23-06ce741095f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:04.897523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df908859-87a7-4d65-9a23-06ce741095f7 None admin] RBAC: Adding query filter params (role.id=46cb171417764f9f9710877b8ec7becc, scope.project.id=47fa4785636148129d33fffc2096cd4b, user.id=dfc58f7c89394e26a4110f493c6fa02d) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:04.897901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df908859-87a7-4d65-9a23-06ce741095f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:04.899334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df908859-87a7-4d65-9a23-06ce741095f7 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:04.912448 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 102/208] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:04 2022] GET /identity/v3/role_assignments?user.id=dfc58f7c89394e26a4110f493c6fa02d&scope.project.id=47fa4785636148129d33fffc2096cd4b&role.id=46cb171417764f9f9710877b8ec7becc => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:06.053068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63f3e94-b334-461e-b1ed-c6b6c8b4ddd8 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:06.054100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63f3e94-b334-461e-b1ed-c6b6c8b4ddd8 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:06.054576 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63f3e94-b334-461e-b1ed-c6b6c8b4ddd8 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:06.056448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 107/209] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:06.061786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af11142c-a043-4478-9b78-1f2fd958e288 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:06.062170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af11142c-a043-4478-9b78-1f2fd958e288 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:06.062343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af11142c-a043-4478-9b78-1f2fd958e288 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:06.080866 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-af11142c-a043-4478-9b78-1f2fd958e288 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:06.082049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-af11142c-a043-4478-9b78-1f2fd958e288 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:06.127624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:06.138719 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa11046a-55dc-4d01-83ac-4d64fbfff6ec None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:06.139009 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa11046a-55dc-4d01-83ac-4d64fbfff6ec None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:06.139273 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa11046a-55dc-4d01-83ac-4d64fbfff6ec None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:06.140646 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 108/210] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:06.149226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f606b7ca-3c6e-4f80-80ce-e5bc0cece11d None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:06.149510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f606b7ca-3c6e-4f80-80ce-e5bc0cece11d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:06.149823 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f606b7ca-3c6e-4f80-80ce-e5bc0cece11d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:06.153699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 103/211] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:06.168412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-f606b7ca-3c6e-4f80-80ce-e5bc0cece11d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:06.169598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f606b7ca-3c6e-4f80-80ce-e5bc0cece11d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:06.243104 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 109/212] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:06.435073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af11142c-a043-4478-9b78-1f2fd958e288 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:06.509278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f606b7ca-3c6e-4f80-80ce-e5bc0cece11d None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:06.555604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af11142c-a043-4478-9b78-1f2fd958e288 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:06.590718 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94643a48-30c2-4b54-8e66-bc45807c3f57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:06.591628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94643a48-30c2-4b54-8e66-bc45807c3f57 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:06.592084 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94643a48-30c2-4b54-8e66-bc45807c3f57 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:06.592528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94643a48-30c2-4b54-8e66-bc45807c3f57 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:06.603883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94643a48-30c2-4b54-8e66-bc45807c3f57 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-94643a48-30c2-4b54-8e66-bc45807c3f57 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:06.608993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:06.610032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:06.610928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:06.610928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:06.610928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:06.610928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:06.610928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:06.610928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:06.610928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:06.610928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:06.610928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:06.610928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:06.610928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:06.610928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:06.612777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 104/213] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:06.618950 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94643a48-30c2-4b54-8e66-bc45807c3f57 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:06.642243 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f606b7ca-3c6e-4f80-80ce-e5bc0cece11d None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:06.651600 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:06.668446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:06.685992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1904d4-364b-4dba-9b2c-92050af9a832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:06.686685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1904d4-364b-4dba-9b2c-92050af9a832 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:06.686875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1904d4-364b-4dba-9b2c-92050af9a832 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:06.687152 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1904d4-364b-4dba-9b2c-92050af9a832 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:06.688270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1904d4-364b-4dba-9b2c-92050af9a832 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:06.689868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1904d4-364b-4dba-9b2c-92050af9a832 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:06.692357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 110/214] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:06.698158 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1904d4-364b-4dba-9b2c-92050af9a832 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:06.741016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94643a48-30c2-4b54-8e66-bc45807c3f57 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:06.786112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0da053f-7b18-455c-8d4a-798557f26db3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:06.787072 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0da053f-7b18-455c-8d4a-798557f26db3 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:06.787356 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0da053f-7b18-455c-8d4a-798557f26db3 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:06.787773 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0da053f-7b18-455c-8d4a-798557f26db3 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:06.788479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0da053f-7b18-455c-8d4a-798557f26db3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:06.788786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0da053f-7b18-455c-8d4a-798557f26db3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:06.790155 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0da053f-7b18-455c-8d4a-798557f26db3 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:06.799108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 105/215] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:06.799108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bdb23b-ab3c-4824-ad91-a40d894174af None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:06.800698 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bdb23b-ab3c-4824-ad91-a40d894174af None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:06.800899 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bdb23b-ab3c-4824-ad91-a40d894174af None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:06.802215 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 106/216] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:06 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:06.806077 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bdb23b-ab3c-4824-ad91-a40d894174af None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:06.834401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1904d4-364b-4dba-9b2c-92050af9a832 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:06.871018 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39440c5b-fa7f-4cff-845a-78611229b02e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:06.871749 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39440c5b-fa7f-4cff-845a-78611229b02e None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:06.872030 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39440c5b-fa7f-4cff-845a-78611229b02e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:06.872326 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39440c5b-fa7f-4cff-845a-78611229b02e None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:06.872988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39440c5b-fa7f-4cff-845a-78611229b02e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:06.874452 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39440c5b-fa7f-4cff-845a-78611229b02e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:06.903892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 111/217] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:01:06 2022] POST /identity/v3/users => generated 297 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:01:06.914856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:06.915946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9535eaa6-1faf-4e68-853b-0a3c6cf228af None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:06.916651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9535eaa6-1faf-4e68-853b-0a3c6cf228af None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:06.917132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9535eaa6-1faf-4e68-853b-0a3c6cf228af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:06.919220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bdb23b-ab3c-4824-ad91-a40d894174af None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:06.944595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-9535eaa6-1faf-4e68-853b-0a3c6cf228af None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:06.946403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9535eaa6-1faf-4e68-853b-0a3c6cf228af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:06.965535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039e3680-a4c2-4a26-a0c6-6ad891417022 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:06.966234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039e3680-a4c2-4a26-a0c6-6ad891417022 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:06.966517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039e3680-a4c2-4a26-a0c6-6ad891417022 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:06.966796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039e3680-a4c2-4a26-a0c6-6ad891417022 None admin] PATH_INFO: `/v3/users/4d5f3c2b84e8401581dbaf7f3bc004b5` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:06.984118 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039e3680-a4c2-4a26-a0c6-6ad891417022 None admin] RBAC: Authorizing `identity:get_user(user_id=4d5f3c2b84e8401581dbaf7f3bc004b5)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:06.985948 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039e3680-a4c2-4a26-a0c6-6ad891417022 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:06.986666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:06.992493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 107/218] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:06 2022] GET /identity/v3/users/4d5f3c2b84e8401581dbaf7f3bc004b5 => generated 297 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:06.994505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039e3680-a4c2-4a26-a0c6-6ad891417022 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:07.058278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 112/219] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:07.130115 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039e3680-a4c2-4a26-a0c6-6ad891417022 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:07.165725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7afb41-c445-4d3b-9897-db82e84c435c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:07.166343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7afb41-c445-4d3b-9897-db82e84c435c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:07.166621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7afb41-c445-4d3b-9897-db82e84c435c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:07.166898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7afb41-c445-4d3b-9897-db82e84c435c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:07.182302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57202332-6e77-4df4-83ff-58ed97443e77 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:07.182581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57202332-6e77-4df4-83ff-58ed97443e77 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:07.182851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57202332-6e77-4df4-83ff-58ed97443e77 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:07.184193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 113/220] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:01:07.189061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01abc29-7f41-4551-ad96-f9a76b2a1515 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:07.189342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01abc29-7f41-4551-ad96-f9a76b2a1515 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:07.189616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01abc29-7f41-4551-ad96-f9a76b2a1515 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:07.197122 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:07.219678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-f01abc29-7f41-4551-ad96-f9a76b2a1515 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:07.221688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f01abc29-7f41-4551-ad96-f9a76b2a1515 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:07.248284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7afb41-c445-4d3b-9897-db82e84c435c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:07.250819 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7afb41-c445-4d3b-9897-db82e84c435c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:07.315154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 114/221] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:07.336506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01abc29-7f41-4551-ad96-f9a76b2a1515 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:07.377511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 108/222] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:07.384860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7afb41-c445-4d3b-9897-db82e84c435c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:07.442220 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:07.447253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:07.455975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01abc29-7f41-4551-ad96-f9a76b2a1515 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:07.491712 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7afb41-c445-4d3b-9897-db82e84c435c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:07.503594 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44eed01b-d297-4568-b1ba-c6bc852ff56a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:07.504671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eed01b-d297-4568-b1ba-c6bc852ff56a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:07.505159 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eed01b-d297-4568-b1ba-c6bc852ff56a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:07.506177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eed01b-d297-4568-b1ba-c6bc852ff56a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:07.518011 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44eed01b-d297-4568-b1ba-c6bc852ff56a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-44eed01b-d297-4568-b1ba-c6bc852ff56a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:07.523966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:07.524885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:07.525660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:07.525660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:07.525660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:07.525660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:07.525660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:07.525660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:07.525660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:07.525660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:07.525660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:07.525660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:07.525660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:01:07.525660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:07.527896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 115/223] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:01:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:07.536782 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac08cb66-ec8d-4168-b633-313aeb810023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:07.536979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44eed01b-d297-4568-b1ba-c6bc852ff56a None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:07.537456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac08cb66-ec8d-4168-b633-313aeb810023 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:07.537734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac08cb66-ec8d-4168-b633-313aeb810023 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:07.538009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac08cb66-ec8d-4168-b633-313aeb810023 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:07.539158 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac08cb66-ec8d-4168-b633-313aeb810023 None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:07.540760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac08cb66-ec8d-4168-b633-313aeb810023 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:07.543314 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 109/224] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:07 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:07.553472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac08cb66-ec8d-4168-b633-313aeb810023 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:07.645668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44eed01b-d297-4568-b1ba-c6bc852ff56a None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:07.683226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878219ad-ccb5-4bac-b365-9175933f66ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:07.684259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878219ad-ccb5-4bac-b365-9175933f66ff None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:07.684913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878219ad-ccb5-4bac-b365-9175933f66ff None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:07.685447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878219ad-ccb5-4bac-b365-9175933f66ff None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:07.686326 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878219ad-ccb5-4bac-b365-9175933f66ff None admin] RBAC: Adding query filter params (name=admin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:07.686839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878219ad-ccb5-4bac-b365-9175933f66ff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:07.688489 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878219ad-ccb5-4bac-b365-9175933f66ff None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:07.698526 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 116/225] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:07.704425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878219ad-ccb5-4bac-b365-9175933f66ff None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:07.724029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:07.742107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac08cb66-ec8d-4168-b633-313aeb810023 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:07.769875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:07.779729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0698b295-f57c-40f2-8c31-764551d8add3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:07.780605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0698b295-f57c-40f2-8c31-764551d8add3 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:07.781028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0698b295-f57c-40f2-8c31-764551d8add3 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:07.781494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0698b295-f57c-40f2-8c31-764551d8add3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:07.782550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0698b295-f57c-40f2-8c31-764551d8add3 None admin] RBAC: Adding query filter params (role.id=615e83fab2504b6298db3efaa7c15750, scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4, user.id=4d5f3c2b84e8401581dbaf7f3bc004b5) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:07.783001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0698b295-f57c-40f2-8c31-764551d8add3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:07.784598 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0698b295-f57c-40f2-8c31-764551d8add3 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:07.791142 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 110/226] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:07 2022] GET /identity/v3/role_assignments?user.id=4d5f3c2b84e8401581dbaf7f3bc004b5&scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4&role.id=615e83fab2504b6298db3efaa7c15750 => generated 259 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:07.792194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0698b295-f57c-40f2-8c31-764551d8add3 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:07.834896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878219ad-ccb5-4bac-b365-9175933f66ff None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:07.879222 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d80c2d-5de1-47b0-91e2-79fd0892ac4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:07.882236 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d80c2d-5de1-47b0-91e2-79fd0892ac4e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:07.882716 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d80c2d-5de1-47b0-91e2-79fd0892ac4e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:07.885733 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d80c2d-5de1-47b0-91e2-79fd0892ac4e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:07.913288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d80c2d-5de1-47b0-91e2-79fd0892ac4e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-36d80c2d-5de1-47b0-91e2-79fd0892ac4e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:07.920153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:07.921261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:07.922190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:07.922190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:07.922190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:07.922190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:07.922190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:07.922190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:07.922190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:07.922190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:07.922190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:07.922190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:07.922190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:07.922190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:07.930628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 117/227] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:01:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:07.942335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d80c2d-5de1-47b0-91e2-79fd0892ac4e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:08.012103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0698b295-f57c-40f2-8c31-764551d8add3 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:08.043254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:08.058460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c38f70-206e-4be8-9d87-6875fac070c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:08.059374 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c38f70-206e-4be8-9d87-6875fac070c3 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:08.059800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c38f70-206e-4be8-9d87-6875fac070c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:08.060220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c38f70-206e-4be8-9d87-6875fac070c3 None admin] PATH_INFO: `/v3/users/1f696d5af6cc481c9ccbd6e4f054f0a1` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:08.061604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c38f70-206e-4be8-9d87-6875fac070c3 None admin] RBAC: Authorizing `identity:get_user(user_id=1f696d5af6cc481c9ccbd6e4f054f0a1)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:08.063303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c38f70-206e-4be8-9d87-6875fac070c3 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:08.065074 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 111/228] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:07 2022] GET /identity/v3/users/1f696d5af6cc481c9ccbd6e4f054f0a1 => generated 283 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:08.071224 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c38f70-206e-4be8-9d87-6875fac070c3 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:08.115682 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d80c2d-5de1-47b0-91e2-79fd0892ac4e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:08.183322 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01e136d-b33c-47e1-8c9b-628e0fc0a33a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:08.184223 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01e136d-b33c-47e1-8c9b-628e0fc0a33a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:08.184669 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01e136d-b33c-47e1-8c9b-628e0fc0a33a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:08.185082 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01e136d-b33c-47e1-8c9b-628e0fc0a33a None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:08.185912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01e136d-b33c-47e1-8c9b-628e0fc0a33a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:08.187294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01e136d-b33c-47e1-8c9b-628e0fc0a33a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:08.189022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01e136d-b33c-47e1-8c9b-628e0fc0a33a None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:08.197497 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:08.201589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 118/229] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:08.209895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01e136d-b33c-47e1-8c9b-628e0fc0a33a None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:08.285717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c38f70-206e-4be8-9d87-6875fac070c3 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:08.343000 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c1632b-4c53-4e26-ac74-c3df8c0443c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:08.343647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c1632b-4c53-4e26-ac74-c3df8c0443c4 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:08.343927 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c1632b-4c53-4e26-ac74-c3df8c0443c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:08.344204 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c1632b-4c53-4e26-ac74-c3df8c0443c4 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:08.346720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c1632b-4c53-4e26-ac74-c3df8c0443c4 None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:08.348167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c1632b-4c53-4e26-ac74-c3df8c0443c4 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:08.353913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 112/230] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:08 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:08.362607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c1632b-4c53-4e26-ac74-c3df8c0443c4 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:08.404059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01e136d-b33c-47e1-8c9b-628e0fc0a33a None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:08.469954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778c0eb9-cdc6-41f8-af08-26ea81a1f715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:08.470597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778c0eb9-cdc6-41f8-af08-26ea81a1f715 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:08.470878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778c0eb9-cdc6-41f8-af08-26ea81a1f715 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:08.472257 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778c0eb9-cdc6-41f8-af08-26ea81a1f715 None admin] PATH_INFO: `/v3/users/1f696d5af6cc481c9ccbd6e4f054f0a1` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:08.474085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778c0eb9-cdc6-41f8-af08-26ea81a1f715 None admin] RBAC: Authorizing `identity:get_user(user_id=1f696d5af6cc481c9ccbd6e4f054f0a1)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:08.476247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778c0eb9-cdc6-41f8-af08-26ea81a1f715 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:08.483198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 119/231] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:08 2022] GET /identity/v3/users/1f696d5af6cc481c9ccbd6e4f054f0a1 => generated 283 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:08.494941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778c0eb9-cdc6-41f8-af08-26ea81a1f715 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:08.496413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:08.549583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c1632b-4c53-4e26-ac74-c3df8c0443c4 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:08.594296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:08.618101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732d4fac-55c7-4f4a-a210-fe6ff38434ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:08.618775 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732d4fac-55c7-4f4a-a210-fe6ff38434ec None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:08.619058 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732d4fac-55c7-4f4a-a210-fe6ff38434ec None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:08.619332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732d4fac-55c7-4f4a-a210-fe6ff38434ec None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4/users/1f696d5af6cc481c9ccbd6e4f054f0a1/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:08.625725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732d4fac-55c7-4f4a-a210-fe6ff38434ec None admin] RBAC: Authorizing `identity:create_grant(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4, user_id=1f696d5af6cc481c9ccbd6e4f054f0a1, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:08.627326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732d4fac-55c7-4f4a-a210-fe6ff38434ec None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:08.645737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 113/232] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:01:08 2022] PUT /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4/users/1f696d5af6cc481c9ccbd6e4f054f0a1/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:01:08.647922 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778c0eb9-cdc6-41f8-af08-26ea81a1f715 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:08.702896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac456b5d-c8a0-4b7c-ab27-89a969340d67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:08.703614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac456b5d-c8a0-4b7c-ab27-89a969340d67 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:08.703893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac456b5d-c8a0-4b7c-ab27-89a969340d67 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:08.704166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac456b5d-c8a0-4b7c-ab27-89a969340d67 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:08.705427 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac456b5d-c8a0-4b7c-ab27-89a969340d67 None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:08.706940 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac456b5d-c8a0-4b7c-ab27-89a969340d67 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:08.709248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 120/233] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:08 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:08.715619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732d4fac-55c7-4f4a-a210-fe6ff38434ec None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:08.858164 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732d4fac-55c7-4f4a-a210-fe6ff38434ec None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:08.895501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ecfb84-9dad-403b-8cf2-d0c8f8837588 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:08.897169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ecfb84-9dad-403b-8cf2-d0c8f8837588 None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:08.897707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ecfb84-9dad-403b-8cf2-d0c8f8837588 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:08.898207 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ecfb84-9dad-403b-8cf2-d0c8f8837588 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4/users/1f696d5af6cc481c9ccbd6e4f054f0a1/roles/a3902a185ac144fb8e24f4e9feb039ed` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:08.900635 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ecfb84-9dad-403b-8cf2-d0c8f8837588 None admin] RBAC: Authorizing `identity:create_grant(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4, user_id=1f696d5af6cc481c9ccbd6e4f054f0a1, role_id=a3902a185ac144fb8e24f4e9feb039ed)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:08.902305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ecfb84-9dad-403b-8cf2-d0c8f8837588 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:08.917770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 114/234] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:01:08 2022] PUT /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4/users/1f696d5af6cc481c9ccbd6e4f054f0a1/roles/a3902a185ac144fb8e24f4e9feb039ed => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:01:09.652137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3588da1a-03f3-47bf-a82a-8d29d5c3c039 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:09.653194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3588da1a-03f3-47bf-a82a-8d29d5c3c039 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:09.653688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3588da1a-03f3-47bf-a82a-8d29d5c3c039 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:09.655484 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 121/235] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:09.661528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3361169e-d930-424c-8b1d-8372d410736a None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:09.663088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3361169e-d930-424c-8b1d-8372d410736a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:09.663570 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3361169e-d930-424c-8b1d-8372d410736a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:09.683680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-3361169e-d930-424c-8b1d-8372d410736a None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:09.685267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-3361169e-d930-424c-8b1d-8372d410736a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:09.736872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:09.764537 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 115/236] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:09.817143 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a972781-c842-41db-9f55-276a659335fe None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:09.817450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a972781-c842-41db-9f55-276a659335fe None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:09.817704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a972781-c842-41db-9f55-276a659335fe None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:09.819597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 122/237] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:09.825009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0df164-9681-4c71-86c8-e0d636652a7d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:09.825295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0df164-9681-4c71-86c8-e0d636652a7d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:09.825572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0df164-9681-4c71-86c8-e0d636652a7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:09.844064 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-dc0df164-9681-4c71-86c8-e0d636652a7d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:09.845294 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-dc0df164-9681-4c71-86c8-e0d636652a7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:09.972447 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 116/238] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:10.097274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a972781-c842-41db-9f55-276a659335fe None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:10.153685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95545d8c-4a8b-469f-97bf-a2697d7c913a None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:10.154005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95545d8c-4a8b-469f-97bf-a2697d7c913a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:10.154236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95545d8c-4a8b-469f-97bf-a2697d7c913a None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:10.155841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 117/239] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:10.164361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c251bdc-716c-4630-b6c6-57a0412351a3 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:10.164361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c251bdc-716c-4630-b6c6-57a0412351a3 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:10.164361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c251bdc-716c-4630-b6c6-57a0412351a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:10.181117 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-6c251bdc-716c-4630-b6c6-57a0412351a3 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:10.182349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6c251bdc-716c-4630-b6c6-57a0412351a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:10.188470 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:10.241055 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a972781-c842-41db-9f55-276a659335fe None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:10.303826 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11164913-8984-4cbc-be2c-b5f125ec1a8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:10.304854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11164913-8984-4cbc-be2c-b5f125ec1a8e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:10.305342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11164913-8984-4cbc-be2c-b5f125ec1a8e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:10.305879 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11164913-8984-4cbc-be2c-b5f125ec1a8e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:10.315646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 118/240] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:10.316673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11164913-8984-4cbc-be2c-b5f125ec1a8e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:10.318915 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11164913-8984-4cbc-be2c-b5f125ec1a8e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:10.320790 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 123/241] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:10.326369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c251bdc-716c-4630-b6c6-57a0412351a3 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:10.379551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:10.388238 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11164913-8984-4cbc-be2c-b5f125ec1a8e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:10.545654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11164913-8984-4cbc-be2c-b5f125ec1a8e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:10.554260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c251bdc-716c-4630-b6c6-57a0412351a3 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:10.602499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440d59bc-e523-42a1-878f-e66394f78a26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:10.604456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d59bc-e523-42a1-878f-e66394f78a26 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:10.604987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d59bc-e523-42a1-878f-e66394f78a26 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:10.605464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d59bc-e523-42a1-878f-e66394f78a26 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:10.616958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:10.619362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148320ca-2b14-419c-ac59-642d5db52fb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:10.620575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148320ca-2b14-419c-ac59-642d5db52fb6 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:10.621848 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148320ca-2b14-419c-ac59-642d5db52fb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:10.622185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440d59bc-e523-42a1-878f-e66394f78a26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:10.622546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148320ca-2b14-419c-ac59-642d5db52fb6 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:10.623484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148320ca-2b14-419c-ac59-642d5db52fb6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:10.625145 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148320ca-2b14-419c-ac59-642d5db52fb6 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-440d59bc-e523-42a1-878f-e66394f78a26 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:10.627814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:10.628703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:10.629488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:10.629488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:10.629488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:10.629488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:10.629488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:10.629488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:10.629488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:10.629488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:10.629488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:10.629488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:10.629488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:01:10.629488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:10.637215 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 124/242] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:10 2022] GET /identity/v3/roles/reader => generated 84 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:10.637893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440d59bc-e523-42a1-878f-e66394f78a26 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:10.657164 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 119/243] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:01:10 2022] POST /identity/v3/users => generated 293 bytes in 331 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:01:10.658552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148320ca-2b14-419c-ac59-642d5db52fb6 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:10.776205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:10.788325 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148320ca-2b14-419c-ac59-642d5db52fb6 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:10.830297 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3df1504-ea51-42ed-a9a4-69d5cc2d6ea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:10.831332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3df1504-ea51-42ed-a9a4-69d5cc2d6ea6 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:10.831819 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3df1504-ea51-42ed-a9a4-69d5cc2d6ea6 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:10.832295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3df1504-ea51-42ed-a9a4-69d5cc2d6ea6 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:10.833464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3df1504-ea51-42ed-a9a4-69d5cc2d6ea6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:10.834554 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3df1504-ea51-42ed-a9a4-69d5cc2d6ea6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:10.837237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3df1504-ea51-42ed-a9a4-69d5cc2d6ea6 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:10.847290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 120/244] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:10 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:10.855335 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3df1504-ea51-42ed-a9a4-69d5cc2d6ea6 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:10.863880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440d59bc-e523-42a1-878f-e66394f78a26 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:10.956036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d847a25d-abb2-4415-b3c1-6b620d8a56c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:10.956407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d847a25d-abb2-4415-b3c1-6b620d8a56c8 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:10.956685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d847a25d-abb2-4415-b3c1-6b620d8a56c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:10.956961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d847a25d-abb2-4415-b3c1-6b620d8a56c8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:10.971494 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d847a25d-abb2-4415-b3c1-6b620d8a56c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-d847a25d-abb2-4415-b3c1-6b620d8a56c8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:10.977039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:10.977946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:10.978813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:10.978813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:10.978813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:10.978813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:10.978813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:10.978813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:10.978813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:10.978813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:10.978813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:10.978813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:10.978813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:10.978813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:10.978813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 125/245] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 341 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:10.984225 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d847a25d-abb2-4415-b3c1-6b620d8a56c8 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:10.996260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3df1504-ea51-42ed-a9a4-69d5cc2d6ea6 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:11.049463 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a84d9c7-09a8-4c1d-a8c0-16108ed7c595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:11.050501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a84d9c7-09a8-4c1d-a8c0-16108ed7c595 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:11.050986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a84d9c7-09a8-4c1d-a8c0-16108ed7c595 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:11.051608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a84d9c7-09a8-4c1d-a8c0-16108ed7c595 None admin] PATH_INFO: `/v3/users/3c2716a4cc604ba4a657172d77928b57` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:11.057949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:11.071821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a84d9c7-09a8-4c1d-a8c0-16108ed7c595 None admin] RBAC: Authorizing `identity:get_user(user_id=3c2716a4cc604ba4a657172d77928b57)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:11.073639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a84d9c7-09a8-4c1d-a8c0-16108ed7c595 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:11.076263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 121/246] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:10 2022] GET /identity/v3/users/3c2716a4cc604ba4a657172d77928b57 => generated 297 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:11.082893 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a84d9c7-09a8-4c1d-a8c0-16108ed7c595 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:11.105510 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:11.120600 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d847a25d-abb2-4415-b3c1-6b620d8a56c8 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:11.157704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c8398f-0673-4ff1-84b1-7c3617c392d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:11.158288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c8398f-0673-4ff1-84b1-7c3617c392d0 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:11.158556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c8398f-0673-4ff1-84b1-7c3617c392d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:11.158831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c8398f-0673-4ff1-84b1-7c3617c392d0 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:11.159570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c8398f-0673-4ff1-84b1-7c3617c392d0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:11.159873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c8398f-0673-4ff1-84b1-7c3617c392d0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:11.161534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c8398f-0673-4ff1-84b1-7c3617c392d0 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:11.172088 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 126/247] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:11.178938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c8398f-0673-4ff1-84b1-7c3617c392d0 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:11.211297 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a84d9c7-09a8-4c1d-a8c0-16108ed7c595 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:11.251299 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ff29d5-fef9-4fbe-aa27-caadbf1f284f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:11.252279 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ff29d5-fef9-4fbe-aa27-caadbf1f284f None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:11.252845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ff29d5-fef9-4fbe-aa27-caadbf1f284f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:11.253320 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ff29d5-fef9-4fbe-aa27-caadbf1f284f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:11.304080 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c8398f-0673-4ff1-84b1-7c3617c392d0 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:11.330342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:11.351468 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4d573e-62e6-4cad-b980-8a0c7212e281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:11.352772 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4d573e-62e6-4cad-b980-8a0c7212e281 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:11.353115 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4d573e-62e6-4cad-b980-8a0c7212e281 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:11.353464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4d573e-62e6-4cad-b980-8a0c7212e281 None admin] PATH_INFO: `/v3/users/4d5f3c2b84e8401581dbaf7f3bc004b5` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:11.354991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4d573e-62e6-4cad-b980-8a0c7212e281 None admin] RBAC: Authorizing `identity:get_user(user_id=4d5f3c2b84e8401581dbaf7f3bc004b5)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:11.355632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ff29d5-fef9-4fbe-aa27-caadbf1f284f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:11.356644 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4d573e-62e6-4cad-b980-8a0c7212e281 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:11.357540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ff29d5-fef9-4fbe-aa27-caadbf1f284f None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:11.358397 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 127/248] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:11 2022] GET /identity/v3/users/4d5f3c2b84e8401581dbaf7f3bc004b5 => generated 297 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:11.363916 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4d573e-62e6-4cad-b980-8a0c7212e281 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:11.376132 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:11.483727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 122/249] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:11.484801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7ee2d3-78c0-48da-bb74-0d5cd7630c66 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:11.485091 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7ee2d3-78c0-48da-bb74-0d5cd7630c66 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:11.485370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7ee2d3-78c0-48da-bb74-0d5cd7630c66 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:11.486956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 123/250] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:11.491206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7ee2d3-78c0-48da-bb74-0d5cd7630c66 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:11.496039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4d573e-62e6-4cad-b980-8a0c7212e281 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:11.536274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4bf4e4-ed38-4839-b281-9ea7e86f6dcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:11.537321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4bf4e4-ed38-4839-b281-9ea7e86f6dcb None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:11.537454 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4bf4e4-ed38-4839-b281-9ea7e86f6dcb None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:11.537818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4bf4e4-ed38-4839-b281-9ea7e86f6dcb None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:11.539029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4bf4e4-ed38-4839-b281-9ea7e86f6dcb None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:11.540617 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4bf4e4-ed38-4839-b281-9ea7e86f6dcb None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:11.542634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 128/251] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:11 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:11.544816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a00d5f5-9b4c-4cf3-8c0e-6cfbe6208fd5 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:11.544816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a00d5f5-9b4c-4cf3-8c0e-6cfbe6208fd5 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:11.544816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a00d5f5-9b4c-4cf3-8c0e-6cfbe6208fd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:11.577369 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-0a00d5f5-9b4c-4cf3-8c0e-6cfbe6208fd5 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:11.579508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0a00d5f5-9b4c-4cf3-8c0e-6cfbe6208fd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:11.654361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7ee2d3-78c0-48da-bb74-0d5cd7630c66 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:11.675152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:11.680761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:11.713976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 129/252] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:11.715393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a00d5f5-9b4c-4cf3-8c0e-6cfbe6208fd5 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:11.717005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae05305-8b1c-4e53-8823-18c8344d201c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:11.717862 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae05305-8b1c-4e53-8823-18c8344d201c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:11.718314 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae05305-8b1c-4e53-8823-18c8344d201c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:11.718757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae05305-8b1c-4e53-8823-18c8344d201c None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:11.721659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae05305-8b1c-4e53-8823-18c8344d201c None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:11.723278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae05305-8b1c-4e53-8823-18c8344d201c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:11.735459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 124/253] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:11 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:11.739121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae05305-8b1c-4e53-8823-18c8344d201c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:11.928292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae05305-8b1c-4e53-8823-18c8344d201c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:11.993128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a00d5f5-9b4c-4cf3-8c0e-6cfbe6208fd5 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:12.034902 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95e16ea-a771-404d-9ada-d1a10d83a95d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:12.035934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95e16ea-a771-404d-9ada-d1a10d83a95d None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:12.036144 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c0c9f1-2647-4996-a928-7a5f6ba19586 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:12.036610 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95e16ea-a771-404d-9ada-d1a10d83a95d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:12.037577 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95e16ea-a771-404d-9ada-d1a10d83a95d None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4/users/4d5f3c2b84e8401581dbaf7f3bc004b5/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:12.040848 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c0c9f1-2647-4996-a928-7a5f6ba19586 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:12.041764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c0c9f1-2647-4996-a928-7a5f6ba19586 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:12.042391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c0c9f1-2647-4996-a928-7a5f6ba19586 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:12.043203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95e16ea-a771-404d-9ada-d1a10d83a95d None admin] RBAC: Authorizing `identity:create_grant(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4, user_id=4d5f3c2b84e8401581dbaf7f3bc004b5, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:12.044108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c0c9f1-2647-4996-a928-7a5f6ba19586 None admin] RBAC: Adding query filter params (role.id=46cb171417764f9f9710877b8ec7becc, scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4, user.id=3c2716a4cc604ba4a657172d77928b57) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:12.045910 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95e16ea-a771-404d-9ada-d1a10d83a95d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:12.046390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c0c9f1-2647-4996-a928-7a5f6ba19586 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:12.047952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c0c9f1-2647-4996-a928-7a5f6ba19586 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:12.059841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 125/254] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:11 2022] GET /identity/v3/role_assignments?user.id=3c2716a4cc604ba4a657172d77928b57&scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4&role.id=46cb171417764f9f9710877b8ec7becc => generated 259 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:12.061490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61282f84-2ca6-439f-8891-c6841391b35a None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:12.067133 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61282f84-2ca6-439f-8891-c6841391b35a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:12.067643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61282f84-2ca6-439f-8891-c6841391b35a None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:12.070325 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 126/255] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:12 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:12.083162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 130/256] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:01:11 2022] PUT /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4/users/4d5f3c2b84e8401581dbaf7f3bc004b5/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:01:12.084804 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203225fc-9350-4b38-bf47-f5a5bb21e893 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:12.085299 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203225fc-9350-4b38-bf47-f5a5bb21e893 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:12.085748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203225fc-9350-4b38-bf47-f5a5bb21e893 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:12.097590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95e16ea-a771-404d-9ada-d1a10d83a95d None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:12.128905 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-203225fc-9350-4b38-bf47-f5a5bb21e893 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:12.130407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-203225fc-9350-4b38-bf47-f5a5bb21e893 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:12.233996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:12.246531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:12.301610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 127/257] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:12.385126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95e16ea-a771-404d-9ada-d1a10d83a95d None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:12.482751 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91938432-8346-41ee-a0e0-4953c9430dbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:12.483486 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91938432-8346-41ee-a0e0-4953c9430dbf None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:12.483777 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91938432-8346-41ee-a0e0-4953c9430dbf None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:12.484049 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91938432-8346-41ee-a0e0-4953c9430dbf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:12.510831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91938432-8346-41ee-a0e0-4953c9430dbf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-91938432-8346-41ee-a0e0-4953c9430dbf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:12.515546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:12.516508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:12.517572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:12.517572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:12.517572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:12.517572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:12.517572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:12.517572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:12.517572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:12.517572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:12.517572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:12.517572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:12.517572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:01:12.517572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:12.517572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 131/258] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:01:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 420 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:12.526306 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203225fc-9350-4b38-bf47-f5a5bb21e893 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:12.645417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203225fc-9350-4b38-bf47-f5a5bb21e893 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:12.694734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382528d2-044e-4893-a8ff-777ea9f596d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:12.695399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382528d2-044e-4893-a8ff-777ea9f596d9 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:12.695683 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382528d2-044e-4893-a8ff-777ea9f596d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:12.695959 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382528d2-044e-4893-a8ff-777ea9f596d9 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:12.696656 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382528d2-044e-4893-a8ff-777ea9f596d9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:12.696963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382528d2-044e-4893-a8ff-777ea9f596d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:12.698337 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382528d2-044e-4893-a8ff-777ea9f596d9 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:12.707620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 128/259] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:12.714109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91938432-8346-41ee-a0e0-4953c9430dbf None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:12.859692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91938432-8346-41ee-a0e0-4953c9430dbf None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:12.867288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:12.911555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b81bb4-2b4c-4511-ba62-8dcb582f0050 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:12.912849 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b81bb4-2b4c-4511-ba62-8dcb582f0050 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:12.913135 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b81bb4-2b4c-4511-ba62-8dcb582f0050 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:12.913411 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b81bb4-2b4c-4511-ba62-8dcb582f0050 None admin] PATH_INFO: `/v3/users/1f696d5af6cc481c9ccbd6e4f054f0a1` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:12.914587 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b81bb4-2b4c-4511-ba62-8dcb582f0050 None admin] RBAC: Authorizing `identity:get_user(user_id=1f696d5af6cc481c9ccbd6e4f054f0a1)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:12.916063 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b81bb4-2b4c-4511-ba62-8dcb582f0050 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:12.918100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 132/260] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:12 2022] GET /identity/v3/users/1f696d5af6cc481c9ccbd6e4f054f0a1 => generated 283 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:12.923589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b81bb4-2b4c-4511-ba62-8dcb582f0050 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:12.923728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382528d2-044e-4893-a8ff-777ea9f596d9 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:12.979702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:13.075251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382528d2-044e-4893-a8ff-777ea9f596d9 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:13.130624 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b81bb4-2b4c-4511-ba62-8dcb582f0050 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:13.132009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8532b78b-d243-4bc7-9278-38c9489a5782 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:13.132661 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8532b78b-d243-4bc7-9278-38c9489a5782 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:13.132938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8532b78b-d243-4bc7-9278-38c9489a5782 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:13.133409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8532b78b-d243-4bc7-9278-38c9489a5782 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:13.253365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8532b78b-d243-4bc7-9278-38c9489a5782 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:13.254838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8532b78b-d243-4bc7-9278-38c9489a5782 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:13.266396 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128c30ef-97ef-4103-8eba-88b1927eaa3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:13.267045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128c30ef-97ef-4103-8eba-88b1927eaa3b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:13.267326 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128c30ef-97ef-4103-8eba-88b1927eaa3b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:13.267602 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128c30ef-97ef-4103-8eba-88b1927eaa3b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:13.301566 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128c30ef-97ef-4103-8eba-88b1927eaa3b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:13.303430 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-128c30ef-97ef-4103-8eba-88b1927eaa3b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:13.306679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:13.308198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:13.309553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:13.309553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:13.309553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:13.309553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:13.309553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:13.309553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:13.309553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:13.309553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:13.309553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:13.309553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:13.309553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:13.309553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:13.316411 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 133/261] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:01:12 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 385 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:13.321325 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128c30ef-97ef-4103-8eba-88b1927eaa3b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:13.392756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:13.418305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 129/262] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 495 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:13.424028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8532b78b-d243-4bc7-9278-38c9489a5782 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:13.518619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128c30ef-97ef-4103-8eba-88b1927eaa3b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:13.569927 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8532b78b-d243-4bc7-9278-38c9489a5782 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:13.617147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acaed425-59ae-40fb-8da8-d3c6653fe68b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:13.617781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acaed425-59ae-40fb-8da8-d3c6653fe68b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:13.618066 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acaed425-59ae-40fb-8da8-d3c6653fe68b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:13.618335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acaed425-59ae-40fb-8da8-d3c6653fe68b None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:13.619024 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acaed425-59ae-40fb-8da8-d3c6653fe68b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:13.619328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acaed425-59ae-40fb-8da8-d3c6653fe68b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:13.621059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acaed425-59ae-40fb-8da8-d3c6653fe68b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:13.623302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61d63aa-5b16-4f3d-8e06-e035b9e3295e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:13.623952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d63aa-5b16-4f3d-8e06-e035b9e3295e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:13.624229 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d63aa-5b16-4f3d-8e06-e035b9e3295e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:13.624527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d63aa-5b16-4f3d-8e06-e035b9e3295e None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:13.629015 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61d63aa-5b16-4f3d-8e06-e035b9e3295e None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:13.630486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61d63aa-5b16-4f3d-8e06-e035b9e3295e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:13.632525 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 130/263] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:13 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:13.635854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 134/264] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:13 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:13.638415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61d63aa-5b16-4f3d-8e06-e035b9e3295e None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:13.641391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acaed425-59ae-40fb-8da8-d3c6653fe68b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:13.652154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:13.785413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61d63aa-5b16-4f3d-8e06-e035b9e3295e None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:13.834113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f1496a-ee93-4a0f-ae5e-76650f03b978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:13.834729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f1496a-ee93-4a0f-ae5e-76650f03b978 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:13.834999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f1496a-ee93-4a0f-ae5e-76650f03b978 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:13.835267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f1496a-ee93-4a0f-ae5e-76650f03b978 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:13.836012 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f1496a-ee93-4a0f-ae5e-76650f03b978 None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4, user.id=1f696d5af6cc481c9ccbd6e4f054f0a1) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:13.836336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f1496a-ee93-4a0f-ae5e-76650f03b978 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:13.838045 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f1496a-ee93-4a0f-ae5e-76650f03b978 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:13.844114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 131/265] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:13 2022] GET /identity/v3/role_assignments?user.id=1f696d5af6cc481c9ccbd6e4f054f0a1&scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 608 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:13.913154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:13.924946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acaed425-59ae-40fb-8da8-d3c6653fe68b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:13.978660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872f9245-d64e-47bb-a0fb-d6b73bbf785c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:13.979305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872f9245-d64e-47bb-a0fb-d6b73bbf785c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:13.979591 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872f9245-d64e-47bb-a0fb-d6b73bbf785c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:13.979866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872f9245-d64e-47bb-a0fb-d6b73bbf785c None admin] PATH_INFO: `/v3/users/1f696d5af6cc481c9ccbd6e4f054f0a1` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:13.981046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872f9245-d64e-47bb-a0fb-d6b73bbf785c None admin] RBAC: Authorizing `identity:get_user(user_id=1f696d5af6cc481c9ccbd6e4f054f0a1)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:13.982544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872f9245-d64e-47bb-a0fb-d6b73bbf785c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:13.984551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 135/266] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:13 2022] GET /identity/v3/users/1f696d5af6cc481c9ccbd6e4f054f0a1 => generated 283 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:13.989841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f1496a-ee93-4a0f-ae5e-76650f03b978 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:14.074680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665f0343-9397-471b-83ee-885545fee039 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:14.075014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665f0343-9397-471b-83ee-885545fee039 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:14.075235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665f0343-9397-471b-83ee-885545fee039 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:14.076922 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 136/267] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:01:14.089851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0690cb0-a7ce-429a-9740-3fb7befe8c74 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:14.090186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0690cb0-a7ce-429a-9740-3fb7befe8c74 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:14.090410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0690cb0-a7ce-429a-9740-3fb7befe8c74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:14.117275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:14.126004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-c0690cb0-a7ce-429a-9740-3fb7befe8c74 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:14.127286 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c0690cb0-a7ce-429a-9740-3fb7befe8c74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:14.180285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f1496a-ee93-4a0f-ae5e-76650f03b978 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:14.243604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34b0d91-b82a-49d6-969b-6f447447e454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:14.244426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34b0d91-b82a-49d6-969b-6f447447e454 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:14.244715 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34b0d91-b82a-49d6-969b-6f447447e454 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:14.244992 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34b0d91-b82a-49d6-969b-6f447447e454 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:14.259730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 137/268] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:14.261038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdcefb4-b51d-4d45-8e07-c134e13b7b63 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:14.261325 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdcefb4-b51d-4d45-8e07-c134e13b7b63 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:14.261598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdcefb4-b51d-4d45-8e07-c134e13b7b63 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:14.263016 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 138/269] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:14.273837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6803cb25-a6e4-49ca-a6d7-fb12ff97b4e4 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:14.274128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6803cb25-a6e4-49ca-a6d7-fb12ff97b4e4 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:14.274396 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6803cb25-a6e4-49ca-a6d7-fb12ff97b4e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:14.309330 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-6803cb25-a6e4-49ca-a6d7-fb12ff97b4e4 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:14.310531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6803cb25-a6e4-49ca-a6d7-fb12ff97b4e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:14.346708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34b0d91-b82a-49d6-969b-6f447447e454 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:14.348200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34b0d91-b82a-49d6-969b-6f447447e454 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:14.433038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 139/270] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:14.455648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:14.488089 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 132/271] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 499 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:14.501207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6803cb25-a6e4-49ca-a6d7-fb12ff97b4e4 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:14.525438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:14.594985 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34b0d91-b82a-49d6-969b-6f447447e454 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:14.643268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6803cb25-a6e4-49ca-a6d7-fb12ff97b4e4 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:14.714623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26346643-b374-4421-bc62-27ceccfa84a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:14.715178 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26346643-b374-4421-bc62-27ceccfa84a6 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:14.715459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26346643-b374-4421-bc62-27ceccfa84a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:14.715730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26346643-b374-4421-bc62-27ceccfa84a6 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:14.716896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26346643-b374-4421-bc62-27ceccfa84a6 None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:14.718350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26346643-b374-4421-bc62-27ceccfa84a6 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:14.719987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 140/272] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:14 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:14.726519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26346643-b374-4421-bc62-27ceccfa84a6 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:14.770945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34b0d91-b82a-49d6-969b-6f447447e454 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:14.809091 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd76e6e8-5f4e-4a2e-98fb-88794c9dbfc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:14.809785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd76e6e8-5f4e-4a2e-98fb-88794c9dbfc7 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:14.810174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd76e6e8-5f4e-4a2e-98fb-88794c9dbfc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:14.810458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd76e6e8-5f4e-4a2e-98fb-88794c9dbfc7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:14.811746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd76e6e8-5f4e-4a2e-98fb-88794c9dbfc7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:14.813218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd76e6e8-5f4e-4a2e-98fb-88794c9dbfc7 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:14.814910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 133/273] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:14.821086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd76e6e8-5f4e-4a2e-98fb-88794c9dbfc7 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:14.864823 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:14.891333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:14.903511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26346643-b374-4421-bc62-27ceccfa84a6 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:14.947812 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd76e6e8-5f4e-4a2e-98fb-88794c9dbfc7 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:14.952201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fdba54-a2d5-4231-832b-4f5a6b77ed01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:14.952950 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fdba54-a2d5-4231-832b-4f5a6b77ed01 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:14.953471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fdba54-a2d5-4231-832b-4f5a6b77ed01 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:14.953951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fdba54-a2d5-4231-832b-4f5a6b77ed01 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:14.954882 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fdba54-a2d5-4231-832b-4f5a6b77ed01 None admin] RBAC: Adding query filter params (role.id=a3902a185ac144fb8e24f4e9feb039ed, scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4, user.id=1f696d5af6cc481c9ccbd6e4f054f0a1) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:14.955407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fdba54-a2d5-4231-832b-4f5a6b77ed01 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:14.958125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fdba54-a2d5-4231-832b-4f5a6b77ed01 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:14.967014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 141/274] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:14 2022] GET /identity/v3/role_assignments?user.id=1f696d5af6cc481c9ccbd6e4f054f0a1&scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4&role.id=a3902a185ac144fb8e24f4e9feb039ed => generated 608 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:14.968555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fdba54-a2d5-4231-832b-4f5a6b77ed01 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:15.013020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a94ac13-a71a-4d0d-8d07-91ac967dd0d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:15.014347 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a94ac13-a71a-4d0d-8d07-91ac967dd0d8 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:15.014821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a94ac13-a71a-4d0d-8d07-91ac967dd0d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:15.015286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a94ac13-a71a-4d0d-8d07-91ac967dd0d8 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:15.016174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a94ac13-a71a-4d0d-8d07-91ac967dd0d8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:15.017860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a94ac13-a71a-4d0d-8d07-91ac967dd0d8 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:15.066202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 134/275] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:01:14 2022] POST /identity/v3/users => generated 293 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:01:15.139563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fdba54-a2d5-4231-832b-4f5a6b77ed01 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:15.190641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2934190-547a-400a-85d8-76260951ec54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:15.191709 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2934190-547a-400a-85d8-76260951ec54 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:15.192268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2934190-547a-400a-85d8-76260951ec54 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:15.192756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2934190-547a-400a-85d8-76260951ec54 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:15.207015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2934190-547a-400a-85d8-76260951ec54 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-a2934190-547a-400a-85d8-76260951ec54 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:15.213563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:15.214618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:15.216059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:15.216059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:15.216059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:15.216059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:15.216059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:15.216059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:15.216059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:15.216059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:15.216059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:15.216059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:15.216059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:15.216059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:15.216059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 142/276] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:14 2022] GET /identity/v3/roles/member => generated 84 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:15.221139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a94ac13-a71a-4d0d-8d07-91ac967dd0d8 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:15.345579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:15.357777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a94ac13-a71a-4d0d-8d07-91ac967dd0d8 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:15.407409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3709dc4-4fd1-453b-878b-3685e00d6a34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:15.408031 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3709dc4-4fd1-453b-878b-3685e00d6a34 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:15.408377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3709dc4-4fd1-453b-878b-3685e00d6a34 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:15.408600 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3709dc4-4fd1-453b-878b-3685e00d6a34 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:15.409281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3709dc4-4fd1-453b-878b-3685e00d6a34 None admin] RBAC: Adding query filter params (name=member) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:15.409584 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3709dc4-4fd1-453b-878b-3685e00d6a34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:15.410942 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3709dc4-4fd1-453b-878b-3685e00d6a34 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:15.422243 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 135/277] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:15.426477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c6bfa-0c16-4bf8-8803-bfce99794280 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:15.426928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c6bfa-0c16-4bf8-8803-bfce99794280 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:15.427028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c6bfa-0c16-4bf8-8803-bfce99794280 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:15.428849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3709dc4-4fd1-453b-878b-3685e00d6a34 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:15.429299 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 143/278] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:15.435499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148cc46d-87b5-4beb-89ea-59eff4e8ec55 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:15.435776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148cc46d-87b5-4beb-89ea-59eff4e8ec55 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:15.436045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148cc46d-87b5-4beb-89ea-59eff4e8ec55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:15.460196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-148cc46d-87b5-4beb-89ea-59eff4e8ec55 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:15.461386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-148cc46d-87b5-4beb-89ea-59eff4e8ec55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:15.478312 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:15.546983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 144/279] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:15.548057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4b9727-91dc-47e9-8e58-8a5d2ba495c6 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:15.548410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4b9727-91dc-47e9-8e58-8a5d2ba495c6 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:15.548684 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4b9727-91dc-47e9-8e58-8a5d2ba495c6 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:15.550064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 145/280] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:15.550571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3709dc4-4fd1-453b-878b-3685e00d6a34 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:15.555916 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050c7c1e-bf34-491e-a1fe-681b12ebef8e None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:15.556199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050c7c1e-bf34-491e-a1fe-681b12ebef8e None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:15.556507 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050c7c1e-bf34-491e-a1fe-681b12ebef8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:15.574840 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-050c7c1e-bf34-491e-a1fe-681b12ebef8e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:15.576035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-050c7c1e-bf34-491e-a1fe-681b12ebef8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:15.591285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c36c1c-8755-49e6-84d0-eb2d41a945b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:15.591928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c36c1c-8755-49e6-84d0-eb2d41a945b1 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:15.592294 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c36c1c-8755-49e6-84d0-eb2d41a945b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:15.592630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c36c1c-8755-49e6-84d0-eb2d41a945b1 None admin] PATH_INFO: `/v3/users/256e5eb65c2e4a6f8ce3c02e31ebb983` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:15.609377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c36c1c-8755-49e6-84d0-eb2d41a945b1 None admin] RBAC: Authorizing `identity:get_user(user_id=256e5eb65c2e4a6f8ce3c02e31ebb983)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:15.610831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c36c1c-8755-49e6-84d0-eb2d41a945b1 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:15.612554 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 136/281] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:15 2022] GET /identity/v3/users/256e5eb65c2e4a6f8ce3c02e31ebb983 => generated 293 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:15.619341 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c36c1c-8755-49e6-84d0-eb2d41a945b1 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:15.635716 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:15.666096 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 146/282] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:15.732529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c36c1c-8755-49e6-84d0-eb2d41a945b1 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:15.769209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938c323e-5a19-4754-8f15-6834bf3090bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:15.770199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938c323e-5a19-4754-8f15-6834bf3090bb None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:15.770695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938c323e-5a19-4754-8f15-6834bf3090bb None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:15.771326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938c323e-5a19-4754-8f15-6834bf3090bb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:15.772323 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938c323e-5a19-4754-8f15-6834bf3090bb None admin] RBAC: Adding query filter params (role.id=615e83fab2504b6298db3efaa7c15750, user.id=256e5eb65c2e4a6f8ce3c02e31ebb983, scope.system=all) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:15.772858 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938c323e-5a19-4754-8f15-6834bf3090bb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:15.774536 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938c323e-5a19-4754-8f15-6834bf3090bb None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:15.781867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 137/283] 10.222.0.26 () {56 vars in 1308 bytes} [Wed Sep 28 12:01:15 2022] GET /identity/v3/role_assignments?user.id=256e5eb65c2e4a6f8ce3c02e31ebb983&scope.system=all&role.id=615e83fab2504b6298db3efaa7c15750 => generated 226 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:15.841502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050c7c1e-bf34-491e-a1fe-681b12ebef8e None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:15.933998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:15.965607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938c323e-5a19-4754-8f15-6834bf3090bb None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:15.966723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050c7c1e-bf34-491e-a1fe-681b12ebef8e None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:16.012417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6702d10b-5b77-454e-9ffe-8fc602a5fc92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:16.013819 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6702d10b-5b77-454e-9ffe-8fc602a5fc92 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:16.014103 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6702d10b-5b77-454e-9ffe-8fc602a5fc92 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:16.014378 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6702d10b-5b77-454e-9ffe-8fc602a5fc92 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:16.027684 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6702d10b-5b77-454e-9ffe-8fc602a5fc92 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-6702d10b-5b77-454e-9ffe-8fc602a5fc92 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:16.033010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:16.033918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:16.034677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:16.034677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:16.034677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:16.034677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:16.034677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:16.034677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:16.034677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:16.034677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:16.034677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:16.034677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:16.034677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:16.034677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:16.034677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 147/284] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:16.040767 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6702d10b-5b77-454e-9ffe-8fc602a5fc92 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:16.103628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:16.134936 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938c323e-5a19-4754-8f15-6834bf3090bb None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:16.201576 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac750a1-b844-4c12-82f5-dfe0224cfe3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:16.202224 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac750a1-b844-4c12-82f5-dfe0224cfe3c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:16.202511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac750a1-b844-4c12-82f5-dfe0224cfe3c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:16.202778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac750a1-b844-4c12-82f5-dfe0224cfe3c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:16.208929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6702d10b-5b77-454e-9ffe-8fc602a5fc92 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:16.218958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac750a1-b844-4c12-82f5-dfe0224cfe3c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-3ac750a1-b844-4c12-82f5-dfe0224cfe3c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:16.226546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:16.227435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:16.228222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:16.228222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:16.228222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:16.228222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:16.228222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:16.228222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:16.228222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:16.228222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:16.228222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:16.228222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:16.228222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:01:16.228222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:16.230784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 138/285] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:15 2022] GET /identity/v3/roles/reader => generated 84 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:16.241537 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac750a1-b844-4c12-82f5-dfe0224cfe3c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:16.251413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42898ec8-5f97-4f4a-9a1f-90c80280f02c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:16.252274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42898ec8-5f97-4f4a-9a1f-90c80280f02c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:16.252744 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42898ec8-5f97-4f4a-9a1f-90c80280f02c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:16.253185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42898ec8-5f97-4f4a-9a1f-90c80280f02c None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:16.253989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42898ec8-5f97-4f4a-9a1f-90c80280f02c None admin] RBAC: Adding query filter params (name=member) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:16.254428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42898ec8-5f97-4f4a-9a1f-90c80280f02c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:16.256615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42898ec8-5f97-4f4a-9a1f-90c80280f02c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:16.266273 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 148/286] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:16.266513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:16.271954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42898ec8-5f97-4f4a-9a1f-90c80280f02c None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:16.396442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac750a1-b844-4c12-82f5-dfe0224cfe3c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:16.461092 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d310f7-3c33-4eb3-99e6-665716ec418a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:16.462934 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d310f7-3c33-4eb3-99e6-665716ec418a None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:16.463224 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d310f7-3c33-4eb3-99e6-665716ec418a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:16.463500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d310f7-3c33-4eb3-99e6-665716ec418a None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:16.464187 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d310f7-3c33-4eb3-99e6-665716ec418a None admin] RBAC: Adding query filter params (name=reader) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:16.464526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d310f7-3c33-4eb3-99e6-665716ec418a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:16.465906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d310f7-3c33-4eb3-99e6-665716ec418a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:16.467402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42898ec8-5f97-4f4a-9a1f-90c80280f02c None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:16.476642 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:16.491490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 139/287] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:16 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:16.501307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d310f7-3c33-4eb3-99e6-665716ec418a None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:16.516455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e9c8b3-3ada-4fbd-a52b-3f179094a34a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:16.517011 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e9c8b3-3ada-4fbd-a52b-3f179094a34a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:16.517294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e9c8b3-3ada-4fbd-a52b-3f179094a34a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:16.517567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e9c8b3-3ada-4fbd-a52b-3f179094a34a None admin] PATH_INFO: `/v3/users/4d5f3c2b84e8401581dbaf7f3bc004b5` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:16.518773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e9c8b3-3ada-4fbd-a52b-3f179094a34a None admin] RBAC: Authorizing `identity:get_user(user_id=4d5f3c2b84e8401581dbaf7f3bc004b5)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:16.520231 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e9c8b3-3ada-4fbd-a52b-3f179094a34a None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:16.522341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 149/288] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:16 2022] GET /identity/v3/users/4d5f3c2b84e8401581dbaf7f3bc004b5 => generated 297 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:16.527795 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e9c8b3-3ada-4fbd-a52b-3f179094a34a None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:16.603043 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:16.617251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d310f7-3c33-4eb3-99e6-665716ec418a None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:16.657690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01de2266-e513-4c29-bea3-a36983fdbc34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:16.658327 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01de2266-e513-4c29-bea3-a36983fdbc34 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:16.658608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01de2266-e513-4c29-bea3-a36983fdbc34 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:16.658884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01de2266-e513-4c29-bea3-a36983fdbc34 None admin] PATH_INFO: `/v3/users/3c2716a4cc604ba4a657172d77928b57` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:16.660179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01de2266-e513-4c29-bea3-a36983fdbc34 None admin] RBAC: Authorizing `identity:get_user(user_id=3c2716a4cc604ba4a657172d77928b57)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:16.661928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01de2266-e513-4c29-bea3-a36983fdbc34 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:16.663941 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 140/289] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:16 2022] GET /identity/v3/users/3c2716a4cc604ba4a657172d77928b57 => generated 297 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:16.671221 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01de2266-e513-4c29-bea3-a36983fdbc34 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:16.672094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e9c8b3-3ada-4fbd-a52b-3f179094a34a None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:16.709215 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ae1179-9f18-4504-aaa2-6f2958b68de6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:16.710204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ae1179-9f18-4504-aaa2-6f2958b68de6 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:16.710739 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ae1179-9f18-4504-aaa2-6f2958b68de6 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:16.711202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ae1179-9f18-4504-aaa2-6f2958b68de6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:16.789728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ae1179-9f18-4504-aaa2-6f2958b68de6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:16.797838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ae1179-9f18-4504-aaa2-6f2958b68de6 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:16.807286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:16.847274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01de2266-e513-4c29-bea3-a36983fdbc34 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:16.888799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf81ea0-a4f9-4893-af0b-4237e3ce6007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:16.889777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf81ea0-a4f9-4893-af0b-4237e3ce6007 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:16.890249 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf81ea0-a4f9-4893-af0b-4237e3ce6007 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:16.890713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf81ea0-a4f9-4893-af0b-4237e3ce6007 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:16.892056 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf81ea0-a4f9-4893-af0b-4237e3ce6007 None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:16.893852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf81ea0-a4f9-4893-af0b-4237e3ce6007 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:16.896757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 141/290] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:16 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:16.902432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf81ea0-a4f9-4893-af0b-4237e3ce6007 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:16.975110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:17.014217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf81ea0-a4f9-4893-af0b-4237e3ce6007 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:17.045312 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 150/291] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 518 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 28 12:01:17.050249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ae1179-9f18-4504-aaa2-6f2958b68de6 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:17.068005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0cfe11-b220-4ac5-8061-bff4118fb53a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:17.068968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0cfe11-b220-4ac5-8061-bff4118fb53a None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:17.069450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0cfe11-b220-4ac5-8061-bff4118fb53a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:17.069897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0cfe11-b220-4ac5-8061-bff4118fb53a None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4/users/3c2716a4cc604ba4a657172d77928b57/roles/46cb171417764f9f9710877b8ec7becc` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:17.073548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0cfe11-b220-4ac5-8061-bff4118fb53a None admin] RBAC: Authorizing `identity:create_grant(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4, user_id=3c2716a4cc604ba4a657172d77928b57, role_id=46cb171417764f9f9710877b8ec7becc)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:17.075190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0cfe11-b220-4ac5-8061-bff4118fb53a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:17.080770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:17.106551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 142/292] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:01:16 2022] PUT /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4/users/3c2716a4cc604ba4a657172d77928b57/roles/46cb171417764f9f9710877b8ec7becc => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:01:17.196062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ae1179-9f18-4504-aaa2-6f2958b68de6 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:17.246920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4837c1aa-8d00-4ed4-89eb-336027066795 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:17.247876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4837c1aa-8d00-4ed4-89eb-336027066795 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:17.248370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4837c1aa-8d00-4ed4-89eb-336027066795 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:17.248858 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4837c1aa-8d00-4ed4-89eb-336027066795 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:17.253115 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4837c1aa-8d00-4ed4-89eb-336027066795 None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:17.254835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4837c1aa-8d00-4ed4-89eb-336027066795 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:17.257154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 151/293] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:17 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:17.262858 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0cfe11-b220-4ac5-8061-bff4118fb53a None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:17.375899 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0cfe11-b220-4ac5-8061-bff4118fb53a None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:17.409756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2f1b69-ae79-41d0-b14e-ecc83ecb612b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:17.410631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2f1b69-ae79-41d0-b14e-ecc83ecb612b None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:17.411088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2f1b69-ae79-41d0-b14e-ecc83ecb612b None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:17.411583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2f1b69-ae79-41d0-b14e-ecc83ecb612b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:17.412551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2f1b69-ae79-41d0-b14e-ecc83ecb612b None admin] RBAC: Adding query filter params (role.id=615e83fab2504b6298db3efaa7c15750, scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4, user.id=4d5f3c2b84e8401581dbaf7f3bc004b5) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:17.413038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2f1b69-ae79-41d0-b14e-ecc83ecb612b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:17.414638 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2f1b69-ae79-41d0-b14e-ecc83ecb612b None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:17.420568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 143/294] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:17 2022] GET /identity/v3/role_assignments?user.id=4d5f3c2b84e8401581dbaf7f3bc004b5&scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4&role.id=615e83fab2504b6298db3efaa7c15750 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:18.088662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d8cc3d-0eee-4023-9dac-7452a3aa7407 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:18.089664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d8cc3d-0eee-4023-9dac-7452a3aa7407 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:18.089947 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d8cc3d-0eee-4023-9dac-7452a3aa7407 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:18.091620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 152/295] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:18.097923 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeafe4f-74c2-4a5b-8c9f-18f66da28166 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:18.098239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeafe4f-74c2-4a5b-8c9f-18f66da28166 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:18.098481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeafe4f-74c2-4a5b-8c9f-18f66da28166 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:18.122552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-cdeafe4f-74c2-4a5b-8c9f-18f66da28166 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:18.123778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-cdeafe4f-74c2-4a5b-8c9f-18f66da28166 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:18.192380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:18.209398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 144/296] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:18.226293 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f592b0-ff00-40c8-ae7d-24f14c2aab82 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:18.226573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f592b0-ff00-40c8-ae7d-24f14c2aab82 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:18.226843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f592b0-ff00-40c8-ae7d-24f14c2aab82 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:18.228666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 153/297] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:18.234192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d592ad2-ddb9-493b-9486-2c377b63a05c None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:18.234723 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d592ad2-ddb9-493b-9486-2c377b63a05c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:18.235198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d592ad2-ddb9-493b-9486-2c377b63a05c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:18.254276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-3d592ad2-ddb9-493b-9486-2c377b63a05c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:18.255834 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-3d592ad2-ddb9-493b-9486-2c377b63a05c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:18.326390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 145/298] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:18.496250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f592b0-ff00-40c8-ae7d-24f14c2aab82 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:18.532485 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:18.544194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a530044-441f-4ab1-b708-0b9fb6659b6e None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:18.544513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a530044-441f-4ab1-b708-0b9fb6659b6e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:18.544791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a530044-441f-4ab1-b708-0b9fb6659b6e None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:18.546405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 146/299] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:01:18.551353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7dffd3-dd12-4cf6-83be-a0a733a6b008 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:18.551636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7dffd3-dd12-4cf6-83be-a0a733a6b008 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:18.551912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7dffd3-dd12-4cf6-83be-a0a733a6b008 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:18.571011 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-0e7dffd3-dd12-4cf6-83be-a0a733a6b008 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:18.572196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0e7dffd3-dd12-4cf6-83be-a0a733a6b008 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:18.607369 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f592b0-ff00-40c8-ae7d-24f14c2aab82 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:18.648478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 147/300] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:18.649417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7dffd3-dd12-4cf6-83be-a0a733a6b008 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:18.655300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4d387f-2286-4261-9777-722e1bce9400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:18.656477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4d387f-2286-4261-9777-722e1bce9400 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:18.656969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4d387f-2286-4261-9777-722e1bce9400 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:18.657441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4d387f-2286-4261-9777-722e1bce9400 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:18.671002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4d387f-2286-4261-9777-722e1bce9400 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-8c4d387f-2286-4261-9777-722e1bce9400 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:18.677810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:18.678688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:18.679649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:18.679649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:18.679649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:18.679649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:18.679649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:18.679649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:18.679649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:18.679649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:18.679649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:18.679649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:18.679649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:01:18.679649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:18.683642 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 154/301] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:18 2022] GET /identity/v3/roles/reader => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:18.697758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4d387f-2286-4261-9777-722e1bce9400 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:18.714012 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:18.768838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7dffd3-dd12-4cf6-83be-a0a733a6b008 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:18.801150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:18.805558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c734a8-00dd-4cdd-92db-beb70a5ce446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:18.806417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c734a8-00dd-4cdd-92db-beb70a5ce446 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:18.806890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c734a8-00dd-4cdd-92db-beb70a5ce446 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:18.807355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c734a8-00dd-4cdd-92db-beb70a5ce446 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:18.808682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c734a8-00dd-4cdd-92db-beb70a5ce446 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:18.808964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4d387f-2286-4261-9777-722e1bce9400 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:18.810533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c734a8-00dd-4cdd-92db-beb70a5ce446 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:18.812276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 148/302] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:18.819764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c734a8-00dd-4cdd-92db-beb70a5ce446 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:18.851237 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ded98e-3565-49c5-8799-c3d4fe7a0e63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:18.851847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ded98e-3565-49c5-8799-c3d4fe7a0e63 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:18.852187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ded98e-3565-49c5-8799-c3d4fe7a0e63 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:18.852661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ded98e-3565-49c5-8799-c3d4fe7a0e63 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:18.853288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ded98e-3565-49c5-8799-c3d4fe7a0e63 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:18.853598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ded98e-3565-49c5-8799-c3d4fe7a0e63 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:18.855041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ded98e-3565-49c5-8799-c3d4fe7a0e63 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:18.868279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 155/303] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:18 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:18.869541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ded98e-3565-49c5-8799-c3d4fe7a0e63 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:18.931232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c734a8-00dd-4cdd-92db-beb70a5ce446 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:18.948797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:18.968297 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b1f3ad-9135-438b-92cc-3db08f808223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:18.968938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b1f3ad-9135-438b-92cc-3db08f808223 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:18.969223 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b1f3ad-9135-438b-92cc-3db08f808223 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:18.969500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b1f3ad-9135-438b-92cc-3db08f808223 None admin] PATH_INFO: `/v3/groups` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:18.970136 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b1f3ad-9135-438b-92cc-3db08f808223 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:18.971565 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b1f3ad-9135-438b-92cc-3db08f808223 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:18.974647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ded98e-3565-49c5-8799-c3d4fe7a0e63 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:18.996337 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 149/304] 10.222.0.26 () {60 vars in 1134 bytes} [Wed Sep 28 12:01:18 2022] POST /identity/v3/groups => generated 229 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:01:18.996337 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b1f3ad-9135-438b-92cc-3db08f808223 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:19.032079 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2b842c-7abb-4be2-b78d-b04d94e1b7fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:19.033344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2b842c-7abb-4be2-b78d-b04d94e1b7fa None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:19.033794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2b842c-7abb-4be2-b78d-b04d94e1b7fa None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:19.034229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2b842c-7abb-4be2-b78d-b04d94e1b7fa None admin] PATH_INFO: `/v3/users/5a8c72fa028f4f479f600b9ac4bdeb3f` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:19.050462 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2b842c-7abb-4be2-b78d-b04d94e1b7fa None admin] RBAC: Authorizing `identity:get_user(user_id=5a8c72fa028f4f479f600b9ac4bdeb3f)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:19.052831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2b842c-7abb-4be2-b78d-b04d94e1b7fa None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:19.055403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 156/305] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:18 2022] GET /identity/v3/users/5a8c72fa028f4f479f600b9ac4bdeb3f => generated 293 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:19.065779 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2b842c-7abb-4be2-b78d-b04d94e1b7fa None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:19.090320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:19.150738 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b1f3ad-9135-438b-92cc-3db08f808223 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:19.194364 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e958a43a-a9f1-4851-919c-f8dec1099983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:19.195097 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e958a43a-a9f1-4851-919c-f8dec1099983 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:19.195536 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e958a43a-a9f1-4851-919c-f8dec1099983 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:19.195898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e958a43a-a9f1-4851-919c-f8dec1099983 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:19.206673 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e958a43a-a9f1-4851-919c-f8dec1099983 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:19.208743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2b842c-7abb-4be2-b78d-b04d94e1b7fa None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-e958a43a-a9f1-4851-919c-f8dec1099983 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:19.212547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:19.213444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:19.219150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:19.219150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:19.219150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:19.219150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:19.219150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:19.219150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:19.219150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:19.219150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:19.219150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:19.219150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:19.219150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:19.219150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:19.219150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 150/306] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:19.221259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e958a43a-a9f1-4851-919c-f8dec1099983 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:19.254103 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7a8792-bfa8-4cda-90e6-9dce806cb3ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:19.255072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7a8792-bfa8-4cda-90e6-9dce806cb3ed None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:19.256496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7a8792-bfa8-4cda-90e6-9dce806cb3ed None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:19.256977 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7a8792-bfa8-4cda-90e6-9dce806cb3ed None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:19.257908 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7a8792-bfa8-4cda-90e6-9dce806cb3ed None admin] RBAC: Adding query filter params (role.id=46cb171417764f9f9710877b8ec7becc, user.id=5a8c72fa028f4f479f600b9ac4bdeb3f, scope.system=all) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:19.258416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7a8792-bfa8-4cda-90e6-9dce806cb3ed None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:19.260081 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7a8792-bfa8-4cda-90e6-9dce806cb3ed None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:19.265803 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 157/307] 10.222.0.26 () {56 vars in 1308 bytes} [Wed Sep 28 12:01:19 2022] GET /identity/v3/role_assignments?user.id=5a8c72fa028f4f479f600b9ac4bdeb3f&scope.system=all&role.id=46cb171417764f9f9710877b8ec7becc => generated 226 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:19.273952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:19.352758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e958a43a-a9f1-4851-919c-f8dec1099983 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:19.400674 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6883d2-dba3-4606-98ba-7fab05511eb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:19.401578 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6883d2-dba3-4606-98ba-7fab05511eb5 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:19.402075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6883d2-dba3-4606-98ba-7fab05511eb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:19.402500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6883d2-dba3-4606-98ba-7fab05511eb5 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:19.403415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6883d2-dba3-4606-98ba-7fab05511eb5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:19.403872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6883d2-dba3-4606-98ba-7fab05511eb5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:19.405467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6883d2-dba3-4606-98ba-7fab05511eb5 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:19.416230 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 151/308] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:19.424429 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7a8792-bfa8-4cda-90e6-9dce806cb3ed None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:19.519557 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:19.543079 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7a8792-bfa8-4cda-90e6-9dce806cb3ed None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:19.579302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d97a35-529f-49ce-9f7e-e99a7036a0f2 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:19.579629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d97a35-529f-49ce-9f7e-e99a7036a0f2 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:19.579741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5639de45-d3ab-418c-b079-627c2618f088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:19.579942 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d97a35-529f-49ce-9f7e-e99a7036a0f2 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:19.580231 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5639de45-d3ab-418c-b079-627c2618f088 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:19.580537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5639de45-d3ab-418c-b079-627c2618f088 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:19.580808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5639de45-d3ab-418c-b079-627c2618f088 None admin] PATH_INFO: `/v3/users/256e5eb65c2e4a6f8ce3c02e31ebb983` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:19.581498 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 152/309] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:01:19.582006 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5639de45-d3ab-418c-b079-627c2618f088 None admin] RBAC: Authorizing `identity:get_user(user_id=256e5eb65c2e4a6f8ce3c02e31ebb983)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:19.583644 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5639de45-d3ab-418c-b079-627c2618f088 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:19.586487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4cbd74-fb3f-4e25-abb3-27cc81561df1 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:19.586781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4cbd74-fb3f-4e25-abb3-27cc81561df1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:19.587315 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4cbd74-fb3f-4e25-abb3-27cc81561df1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:19.588389 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 158/310] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:19 2022] GET /identity/v3/users/256e5eb65c2e4a6f8ce3c02e31ebb983 => generated 293 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:19.591803 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5639de45-d3ab-418c-b079-627c2618f088 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:19.606482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-aa4cbd74-fb3f-4e25-abb3-27cc81561df1 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:19.607595 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-aa4cbd74-fb3f-4e25-abb3-27cc81561df1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:19.679147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 153/311] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:19.707291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5639de45-d3ab-418c-b079-627c2618f088 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:19.752857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cb0608-6e94-4c66-92eb-4373ef33bf72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:19.753890 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cb0608-6e94-4c66-92eb-4373ef33bf72 None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:19.754370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cb0608-6e94-4c66-92eb-4373ef33bf72 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:19.754884 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cb0608-6e94-4c66-92eb-4373ef33bf72 None admin] PATH_INFO: `/v3/system/users/256e5eb65c2e4a6f8ce3c02e31ebb983/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:19.757153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cb0608-6e94-4c66-92eb-4373ef33bf72 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=256e5eb65c2e4a6f8ce3c02e31ebb983, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:19.758730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cb0608-6e94-4c66-92eb-4373ef33bf72 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:19.770204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 159/312] 10.222.0.26 () {58 vars in 1334 bytes} [Wed Sep 28 12:01:19 2022] PUT /identity/v3/system/users/256e5eb65c2e4a6f8ce3c02e31ebb983/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:01:19.979124 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4cbd74-fb3f-4e25-abb3-27cc81561df1 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:20.008983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:20.098275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4cbd74-fb3f-4e25-abb3-27cc81561df1 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:20.135888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba647d5-ddd2-47c7-b01b-77f33533503f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:20.136944 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba647d5-ddd2-47c7-b01b-77f33533503f None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:20.137454 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba647d5-ddd2-47c7-b01b-77f33533503f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:20.137983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba647d5-ddd2-47c7-b01b-77f33533503f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:20.149538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba647d5-ddd2-47c7-b01b-77f33533503f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-0ba647d5-ddd2-47c7-b01b-77f33533503f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:20.154706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:20.155969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:20.157113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:20.157113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:20.157113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:20.157113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:20.157113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:20.157113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:20.157113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:20.157113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:20.157113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:20.157113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:20.157113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:01:20.157113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:20.159420 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 154/313] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:19 2022] GET /identity/v3/roles/reader => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:20.165907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cb0608-6e94-4c66-92eb-4373ef33bf72 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:20.179475 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:20.282387 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cb0608-6e94-4c66-92eb-4373ef33bf72 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:20.322332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe02e7f-0802-4a74-82fc-c7f257d9df4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:20.322938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe02e7f-0802-4a74-82fc-c7f257d9df4a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:20.323218 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe02e7f-0802-4a74-82fc-c7f257d9df4a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:20.323491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe02e7f-0802-4a74-82fc-c7f257d9df4a None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:20.324155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe02e7f-0802-4a74-82fc-c7f257d9df4a None admin] RBAC: Adding query filter params (name=reader) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:20.324561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe02e7f-0802-4a74-82fc-c7f257d9df4a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:20.325958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe02e7f-0802-4a74-82fc-c7f257d9df4a None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:20.335096 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 160/314] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:20 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:20.340996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba647d5-ddd2-47c7-b01b-77f33533503f None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:20.434352 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:20.448985 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba647d5-ddd2-47c7-b01b-77f33533503f None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:20.486429 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9006c5-c8e5-48af-9ed4-c725f16a37c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:20.487438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9006c5-c8e5-48af-9ed4-c725f16a37c0 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:20.487920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9006c5-c8e5-48af-9ed4-c725f16a37c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:20.488448 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9006c5-c8e5-48af-9ed4-c725f16a37c0 None admin] PATH_INFO: `/v3/users/3c2716a4cc604ba4a657172d77928b57` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:20.490333 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9006c5-c8e5-48af-9ed4-c725f16a37c0 None admin] RBAC: Authorizing `identity:get_user(user_id=3c2716a4cc604ba4a657172d77928b57)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:20.492008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9006c5-c8e5-48af-9ed4-c725f16a37c0 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:20.494163 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 155/315] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:20 2022] GET /identity/v3/users/3c2716a4cc604ba4a657172d77928b57 => generated 297 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:20.499988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe02e7f-0802-4a74-82fc-c7f257d9df4a None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:20.573189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:20.603954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe02e7f-0802-4a74-82fc-c7f257d9df4a None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:20.638627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc0b79f-7a75-4116-aaf6-be1f1c424768 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:20.639238 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc0b79f-7a75-4116-aaf6-be1f1c424768 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:20.639517 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc0b79f-7a75-4116-aaf6-be1f1c424768 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:20.639793 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc0b79f-7a75-4116-aaf6-be1f1c424768 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:20.712660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc0b79f-7a75-4116-aaf6-be1f1c424768 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:20.714141 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc0b79f-7a75-4116-aaf6-be1f1c424768 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:20.780750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:20.812088 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 161/316] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:20.817426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9006c5-c8e5-48af-9ed4-c725f16a37c0 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:20.922032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9006c5-c8e5-48af-9ed4-c725f16a37c0 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:20.957385 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed2a6e7-edff-4c1a-84e9-9c7627817e27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:20.958016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed2a6e7-edff-4c1a-84e9-9c7627817e27 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:20.958349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed2a6e7-edff-4c1a-84e9-9c7627817e27 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:20.958642 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed2a6e7-edff-4c1a-84e9-9c7627817e27 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:20.959790 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed2a6e7-edff-4c1a-84e9-9c7627817e27 None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:20.961260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed2a6e7-edff-4c1a-84e9-9c7627817e27 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:20.963453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 156/317] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:20 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:20.973428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc0b79f-7a75-4116-aaf6-be1f1c424768 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:21.075824 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc0b79f-7a75-4116-aaf6-be1f1c424768 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:21.118920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f0f51b-9a3f-4fe8-8e16-e729b8574aa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:21.120084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f0f51b-9a3f-4fe8-8e16-e729b8574aa7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:21.120084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f0f51b-9a3f-4fe8-8e16-e729b8574aa7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:21.120084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f0f51b-9a3f-4fe8-8e16-e729b8574aa7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:21.120870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f0f51b-9a3f-4fe8-8e16-e729b8574aa7 None admin] RBAC: Adding query filter params (role.id=46cb171417764f9f9710877b8ec7becc, scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4, user.id=3c2716a4cc604ba4a657172d77928b57) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:21.121422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f0f51b-9a3f-4fe8-8e16-e729b8574aa7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:21.122873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f0f51b-9a3f-4fe8-8e16-e729b8574aa7 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:21.129721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 162/318] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:20 2022] GET /identity/v3/role_assignments?user.id=3c2716a4cc604ba4a657172d77928b57&scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4&role.id=46cb171417764f9f9710877b8ec7becc => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:21.457357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ddd413-2fa8-49c3-af59-e520a6a840d1 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:21.457906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ddd413-2fa8-49c3-af59-e520a6a840d1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:21.457906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ddd413-2fa8-49c3-af59-e520a6a840d1 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:21.459545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 157/319] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:21.465032 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b888c5-a79b-471d-8642-b90607a8982b None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:21.465421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b888c5-a79b-471d-8642-b90607a8982b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:21.465580 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b888c5-a79b-471d-8642-b90607a8982b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:21.483664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-00b888c5-a79b-471d-8642-b90607a8982b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:21.484891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-00b888c5-a79b-471d-8642-b90607a8982b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:21.515620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:21.518523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a553524f-2064-45b0-8709-f4f8c134dace None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:21.518941 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a553524f-2064-45b0-8709-f4f8c134dace None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:21.519221 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a553524f-2064-45b0-8709-f4f8c134dace None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:21.521096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 158/320] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:21.525826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7af26e-1bda-4a03-9a04-56b4ff023115 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:21.526252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7af26e-1bda-4a03-9a04-56b4ff023115 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:21.526417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7af26e-1bda-4a03-9a04-56b4ff023115 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:21.544295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4d7af26e-1bda-4a03-9a04-56b4ff023115 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:21.545509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4d7af26e-1bda-4a03-9a04-56b4ff023115 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:21.564188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:21.565831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 163/321] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:21.615819 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 159/322] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:21.830510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b888c5-a79b-471d-8642-b90607a8982b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:21.882528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7af26e-1bda-4a03-9a04-56b4ff023115 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:21.940553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b888c5-a79b-471d-8642-b90607a8982b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:21.980263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2035925-713e-4370-b9bc-83baa9c8e079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:21.981292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2035925-713e-4370-b9bc-83baa9c8e079 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:21.981770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2035925-713e-4370-b9bc-83baa9c8e079 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:21.982064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2035925-713e-4370-b9bc-83baa9c8e079 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:21.983233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2035925-713e-4370-b9bc-83baa9c8e079 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:21.984760 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2035925-713e-4370-b9bc-83baa9c8e079 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:21.986785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 164/323] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:21.992493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2035925-713e-4370-b9bc-83baa9c8e079 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:21.992881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7af26e-1bda-4a03-9a04-56b4ff023115 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:22.025913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:22.026066 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5241b81-531d-4edd-894d-40da715df944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:22.026541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5241b81-531d-4edd-894d-40da715df944 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:22.026824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5241b81-531d-4edd-894d-40da715df944 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:22.027108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5241b81-531d-4edd-894d-40da715df944 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:22.036950 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5241b81-531d-4edd-894d-40da715df944 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-a5241b81-531d-4edd-894d-40da715df944 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:22.041722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:22.042617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:22.043457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:22.043457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:22.043457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:22.043457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:22.043457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:22.043457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:22.043457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:22.043457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:22.043457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:22.043457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:22.043457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:01:22.043457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:22.043457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 160/324] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:21 2022] GET /identity/v3/roles/reader => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:22.044143 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec733736-8cb5-4cd0-9432-5a0a43888663 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:22.044206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec733736-8cb5-4cd0-9432-5a0a43888663 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:22.044806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec733736-8cb5-4cd0-9432-5a0a43888663 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:22.045715 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 161/325] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:22 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:22.049225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec733736-8cb5-4cd0-9432-5a0a43888663 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:22.065069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:22.094490 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2035925-713e-4370-b9bc-83baa9c8e079 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:22.128787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3199d8eb-6ae0-49f9-bfc6-312ba567a30d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:22.129387 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3199d8eb-6ae0-49f9-bfc6-312ba567a30d None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:22.129664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3199d8eb-6ae0-49f9-bfc6-312ba567a30d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:22.130252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3199d8eb-6ae0-49f9-bfc6-312ba567a30d None admin] PATH_INFO: `/v3/groups` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:22.130889 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3199d8eb-6ae0-49f9-bfc6-312ba567a30d None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:22.132379 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3199d8eb-6ae0-49f9-bfc6-312ba567a30d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:22.147818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 165/326] 10.222.0.26 () {60 vars in 1134 bytes} [Wed Sep 28 12:01:21 2022] POST /identity/v3/groups => generated 226 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:01:22.149031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c98d1f1-a7d5-4bcc-b138-8196a9546da5 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:22.149318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c98d1f1-a7d5-4bcc-b138-8196a9546da5 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:22.149586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c98d1f1-a7d5-4bcc-b138-8196a9546da5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:22.156840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec733736-8cb5-4cd0-9432-5a0a43888663 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:22.173040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-6c98d1f1-a7d5-4bcc-b138-8196a9546da5 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:22.174268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6c98d1f1-a7d5-4bcc-b138-8196a9546da5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:22.198130 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa0cb40-d84f-4caf-9fcb-5da297825c45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:22.198761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa0cb40-d84f-4caf-9fcb-5da297825c45 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:22.199041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa0cb40-d84f-4caf-9fcb-5da297825c45 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:22.199312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa0cb40-d84f-4caf-9fcb-5da297825c45 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:22.199999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa0cb40-d84f-4caf-9fcb-5da297825c45 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:22.200344 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa0cb40-d84f-4caf-9fcb-5da297825c45 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:22.201774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa0cb40-d84f-4caf-9fcb-5da297825c45 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:22.212639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 162/327] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:22 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:22.217938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa0cb40-d84f-4caf-9fcb-5da297825c45 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:22.257258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 166/328] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:22.315917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:22.343950 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa0cb40-d84f-4caf-9fcb-5da297825c45 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:22.378231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edad5508-9674-4218-9926-260a69252e48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:22.378784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edad5508-9674-4218-9926-260a69252e48 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:22.379062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edad5508-9674-4218-9926-260a69252e48 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:22.379333 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edad5508-9674-4218-9926-260a69252e48 None admin] PATH_INFO: `/v3/users/5a8c72fa028f4f479f600b9ac4bdeb3f` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:22.381008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edad5508-9674-4218-9926-260a69252e48 None admin] RBAC: Authorizing `identity:get_user(user_id=5a8c72fa028f4f479f600b9ac4bdeb3f)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:22.382432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edad5508-9674-4218-9926-260a69252e48 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:22.384037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 163/329] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:22 2022] GET /identity/v3/users/5a8c72fa028f4f479f600b9ac4bdeb3f => generated 293 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:22.389677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c98d1f1-a7d5-4bcc-b138-8196a9546da5 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:22.423761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:22.510942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c98d1f1-a7d5-4bcc-b138-8196a9546da5 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:22.554044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe4e779-044d-4c44-ba53-390f30dcde8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:22.554696 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe4e779-044d-4c44-ba53-390f30dcde8b None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:22.554975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe4e779-044d-4c44-ba53-390f30dcde8b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:22.555248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe4e779-044d-4c44-ba53-390f30dcde8b None admin] PATH_INFO: `/v3/system/users/5a8c72fa028f4f479f600b9ac4bdeb3f/roles/46cb171417764f9f9710877b8ec7becc` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:22.556780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe4e779-044d-4c44-ba53-390f30dcde8b None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=5a8c72fa028f4f479f600b9ac4bdeb3f, role_id=46cb171417764f9f9710877b8ec7becc)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:22.558217 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe4e779-044d-4c44-ba53-390f30dcde8b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:22.567428 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edad5508-9674-4218-9926-260a69252e48 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:22.568465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 167/330] 10.222.0.26 () {58 vars in 1334 bytes} [Wed Sep 28 12:01:22 2022] PUT /identity/v3/system/users/5a8c72fa028f4f479f600b9ac4bdeb3f/roles/46cb171417764f9f9710877b8ec7becc => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:01:22.696202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edad5508-9674-4218-9926-260a69252e48 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:22.734161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139762e8-ebc6-40a7-a0a3-e141d1ee11fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:22.734808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139762e8-ebc6-40a7-a0a3-e141d1ee11fe None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:22.735090 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139762e8-ebc6-40a7-a0a3-e141d1ee11fe None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:22.735358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139762e8-ebc6-40a7-a0a3-e141d1ee11fe None admin] PATH_INFO: `/v3/roles/member` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:22.747614 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139762e8-ebc6-40a7-a0a3-e141d1ee11fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-139762e8-ebc6-40a7-a0a3-e141d1ee11fe None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:22.754270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:22.755274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:22.756606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:22.756606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:22.756606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:22.756606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:22.756606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:22.756606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:22.756606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:22.756606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:22.756606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:22.756606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:22.756606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:22.756606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:22.756606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 164/331] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:22.762297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe4e779-044d-4c44-ba53-390f30dcde8b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:22.866618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:22.880815 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe4e779-044d-4c44-ba53-390f30dcde8b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:22.917168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a1801a-ddaa-45bb-b3d4-aa594126ec81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:22.917776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a1801a-ddaa-45bb-b3d4-aa594126ec81 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:22.918052 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a1801a-ddaa-45bb-b3d4-aa594126ec81 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:22.918320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a1801a-ddaa-45bb-b3d4-aa594126ec81 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:22.918984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a1801a-ddaa-45bb-b3d4-aa594126ec81 None admin] RBAC: Adding query filter params (name=member) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:22.919280 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a1801a-ddaa-45bb-b3d4-aa594126ec81 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:22.920799 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a1801a-ddaa-45bb-b3d4-aa594126ec81 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:22.932450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 168/332] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:22.935400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139762e8-ebc6-40a7-a0a3-e141d1ee11fe None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:22.945878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:23.045770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139762e8-ebc6-40a7-a0a3-e141d1ee11fe None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:23.089990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89c0db9-2693-444c-b8f4-46558e14172e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:23.090634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89c0db9-2693-444c-b8f4-46558e14172e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:23.090914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89c0db9-2693-444c-b8f4-46558e14172e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:23.091188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89c0db9-2693-444c-b8f4-46558e14172e None admin] PATH_INFO: `/v3/users/256e5eb65c2e4a6f8ce3c02e31ebb983` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:23.092360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89c0db9-2693-444c-b8f4-46558e14172e None admin] RBAC: Authorizing `identity:get_user(user_id=256e5eb65c2e4a6f8ce3c02e31ebb983)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:23.093799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89c0db9-2693-444c-b8f4-46558e14172e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:23.095691 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 165/333] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:22 2022] GET /identity/v3/users/256e5eb65c2e4a6f8ce3c02e31ebb983 => generated 293 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:23.102111 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a1801a-ddaa-45bb-b3d4-aa594126ec81 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:23.231652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a1801a-ddaa-45bb-b3d4-aa594126ec81 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:23.282121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8aa6d86-b1a9-4ec4-9a3b-555226fe35b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:23.282997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8aa6d86-b1a9-4ec4-9a3b-555226fe35b4 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:23.283451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8aa6d86-b1a9-4ec4-9a3b-555226fe35b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:23.283855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8aa6d86-b1a9-4ec4-9a3b-555226fe35b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:23.284753 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8aa6d86-b1a9-4ec4-9a3b-555226fe35b4 None admin] RBAC: Adding query filter params (role.id=615e83fab2504b6298db3efaa7c15750, user.id=256e5eb65c2e4a6f8ce3c02e31ebb983, scope.system=all) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:23.285274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8aa6d86-b1a9-4ec4-9a3b-555226fe35b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:23.287064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8aa6d86-b1a9-4ec4-9a3b-555226fe35b4 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:23.301253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 169/334] 10.222.0.26 () {56 vars in 1308 bytes} [Wed Sep 28 12:01:23 2022] GET /identity/v3/role_assignments?user.id=256e5eb65c2e4a6f8ce3c02e31ebb983&scope.system=all&role.id=615e83fab2504b6298db3efaa7c15750 => generated 510 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:24.989684 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593ba97b-3085-466c-b623-5898aa0b768e None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:24.990222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593ba97b-3085-466c-b623-5898aa0b768e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:24.990222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593ba97b-3085-466c-b623-5898aa0b768e None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:24.991905 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 166/335] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:24.997197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42925c8e-c348-4389-ab1f-38478e571bba None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:24.997626 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42925c8e-c348-4389-ab1f-38478e571bba None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:24.997817 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42925c8e-c348-4389-ab1f-38478e571bba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:25.025096 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-42925c8e-c348-4389-ab1f-38478e571bba None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:25.026298 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-42925c8e-c348-4389-ab1f-38478e571bba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:25.028426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:25.114807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 170/336] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:25.135857 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2916f8a6-1df1-4c25-8b1b-e0fa0c9fc803 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:25.136187 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2916f8a6-1df1-4c25-8b1b-e0fa0c9fc803 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:25.136460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2916f8a6-1df1-4c25-8b1b-e0fa0c9fc803 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:25.137806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 167/337] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:25.148711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103412d8-7a4d-4791-974f-4d562fc03c02 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:25.149012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103412d8-7a4d-4791-974f-4d562fc03c02 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:25.149277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103412d8-7a4d-4791-974f-4d562fc03c02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:25.175749 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-103412d8-7a4d-4791-974f-4d562fc03c02 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:25.177049 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-103412d8-7a4d-4791-974f-4d562fc03c02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:25.259002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 171/338] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:25.340143 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11f7a62-bfe8-4403-907a-22861b6ac7eb None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:25.340542 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11f7a62-bfe8-4403-907a-22861b6ac7eb None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:25.340744 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11f7a62-bfe8-4403-907a-22861b6ac7eb None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:25.342351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 168/339] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:25.347782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c2c188-47f9-4cc0-bebd-34b61e552737 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:25.348073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c2c188-47f9-4cc0-bebd-34b61e552737 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:25.348372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c2c188-47f9-4cc0-bebd-34b61e552737 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:25.375877 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-47c2c188-47f9-4cc0-bebd-34b61e552737 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:25.377106 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-47c2c188-47f9-4cc0-bebd-34b61e552737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:25.449209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:25.457966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 172/340] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:25.507074 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11f7a62-bfe8-4403-907a-22861b6ac7eb None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:25.596529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:25.633183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11f7a62-bfe8-4403-907a-22861b6ac7eb None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:25.674967 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0af5f5-d8f3-4c3f-adfd-41715844bd93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:25.675601 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0af5f5-d8f3-4c3f-adfd-41715844bd93 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:25.675876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0af5f5-d8f3-4c3f-adfd-41715844bd93 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:25.676142 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0af5f5-d8f3-4c3f-adfd-41715844bd93 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:25.686124 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c2c188-47f9-4cc0-bebd-34b61e552737 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:25.688869 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0af5f5-d8f3-4c3f-adfd-41715844bd93 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-4f0af5f5-d8f3-4c3f-adfd-41715844bd93 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:25.693710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:25.694585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:25.695369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:25.695369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:25.695369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:25.695369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:25.695369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:25.695369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:25.695369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:25.695369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:25.695369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:25.695369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:25.695369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:25.695369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:25.695369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 169/341] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:25 2022] GET /identity/v3/roles/member => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:25.701129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0af5f5-d8f3-4c3f-adfd-41715844bd93 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:25.820300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c2c188-47f9-4cc0-bebd-34b61e552737 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:25.879703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56aa16e9-9d97-46bc-9f8e-6d0871b76fb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:25.880335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56aa16e9-9d97-46bc-9f8e-6d0871b76fb1 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:25.880608 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56aa16e9-9d97-46bc-9f8e-6d0871b76fb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:25.880880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56aa16e9-9d97-46bc-9f8e-6d0871b76fb1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:25.899793 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56aa16e9-9d97-46bc-9f8e-6d0871b76fb1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-56aa16e9-9d97-46bc-9f8e-6d0871b76fb1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:25.905034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:25.905898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:25.906671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:25.906671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:25.906671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:25.906671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:25.906671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:25.906671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:25.906671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:25.906671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:25.906671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:25.906671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:25.906671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 28 12:01:25.906671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:25.908480 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0af5f5-d8f3-4c3f-adfd-41715844bd93 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:25.909865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 173/342] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:25 2022] GET /identity/v3/roles/reader => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:25.930183 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56aa16e9-9d97-46bc-9f8e-6d0871b76fb1 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:25.953228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:25.969582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ea8e52-8928-4717-8d28-bd15aec023d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:25.970192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ea8e52-8928-4717-8d28-bd15aec023d8 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:25.970466 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ea8e52-8928-4717-8d28-bd15aec023d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:25.970731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ea8e52-8928-4717-8d28-bd15aec023d8 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:25.971397 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ea8e52-8928-4717-8d28-bd15aec023d8 None admin] RBAC: Adding query filter params (name=member) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:25.971695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ea8e52-8928-4717-8d28-bd15aec023d8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:25.973079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ea8e52-8928-4717-8d28-bd15aec023d8 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:25.986631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 170/343] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:25.993852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ea8e52-8928-4717-8d28-bd15aec023d8 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:26.031179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:26.115179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56aa16e9-9d97-46bc-9f8e-6d0871b76fb1 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:26.145622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ea8e52-8928-4717-8d28-bd15aec023d8 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:26.179594 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba36baf-f60e-4fd1-b398-72b83afb5aca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:26.180204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba36baf-f60e-4fd1-b398-72b83afb5aca None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:26.180502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba36baf-f60e-4fd1-b398-72b83afb5aca None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:26.180867 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba36baf-f60e-4fd1-b398-72b83afb5aca None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:26.181649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba36baf-f60e-4fd1-b398-72b83afb5aca None admin] RBAC: Adding query filter params (name=reader) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:26.181949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba36baf-f60e-4fd1-b398-72b83afb5aca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:26.183499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba36baf-f60e-4fd1-b398-72b83afb5aca None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:26.188681 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c170e3-9b46-496a-9b26-5104294afd32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:26.189288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c170e3-9b46-496a-9b26-5104294afd32 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:26.189564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c170e3-9b46-496a-9b26-5104294afd32 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:26.189840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c170e3-9b46-496a-9b26-5104294afd32 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:26.194109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 174/344] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:25 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:26.195534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb34a787-34b9-455b-ab7d-4267162f46b5 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:26.195989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb34a787-34b9-455b-ab7d-4267162f46b5 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:26.196519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb34a787-34b9-455b-ab7d-4267162f46b5 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:26.198044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 175/345] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:26.199001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb34a787-34b9-455b-ab7d-4267162f46b5 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:26.271562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:26.281448 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c170e3-9b46-496a-9b26-5104294afd32 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:26.281887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c170e3-9b46-496a-9b26-5104294afd32 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:26.347535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb34a787-34b9-455b-ab7d-4267162f46b5 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:26.375809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:26.397778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030cf9ab-f107-49ba-8c78-ac935403e34b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:26.398957 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030cf9ab-f107-49ba-8c78-ac935403e34b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:26.399654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030cf9ab-f107-49ba-8c78-ac935403e34b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:26.400124 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030cf9ab-f107-49ba-8c78-ac935403e34b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:26.415278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030cf9ab-f107-49ba-8c78-ac935403e34b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:26.421627 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 171/346] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 28 12:01:26.422587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c170e3-9b46-496a-9b26-5104294afd32 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-030cf9ab-f107-49ba-8c78-ac935403e34b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:26.422827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:26.423775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:26.424654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:26.424654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:26.424654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:26.424654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:26.424654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:26.424654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:26.424654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:26.424654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:26.424654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:26.424654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:26.424654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:26.424654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:26.425575 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 176/347] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:01:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:26.426768 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83544234-4f1d-4e45-ba56-cceb1a8c2e73 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:26.427656 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83544234-4f1d-4e45-ba56-cceb1a8c2e73 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:26.427955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83544234-4f1d-4e45-ba56-cceb1a8c2e73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:26.450397 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-83544234-4f1d-4e45-ba56-cceb1a8c2e73 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:26.451607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-83544234-4f1d-4e45-ba56-cceb1a8c2e73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:26.532006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:26.538559 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 177/348] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:26.539404 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83544234-4f1d-4e45-ba56-cceb1a8c2e73 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:26.540946 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c170e3-9b46-496a-9b26-5104294afd32 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:26.580594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34bb01b-9bd3-478f-b8a3-b55ada6a4daa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:26.581535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34bb01b-9bd3-478f-b8a3-b55ada6a4daa None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:26.582014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34bb01b-9bd3-478f-b8a3-b55ada6a4daa None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:26.582460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34bb01b-9bd3-478f-b8a3-b55ada6a4daa None admin] PATH_INFO: `/v3/users/5a8c72fa028f4f479f600b9ac4bdeb3f` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:26.585055 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34bb01b-9bd3-478f-b8a3-b55ada6a4daa None admin] RBAC: Authorizing `identity:get_user(user_id=5a8c72fa028f4f479f600b9ac4bdeb3f)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:26.586665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34bb01b-9bd3-478f-b8a3-b55ada6a4daa None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:26.588962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 172/349] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:01:26 2022] GET /identity/v3/users/5a8c72fa028f4f479f600b9ac4bdeb3f => generated 293 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:26.589811 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34bb01b-9bd3-478f-b8a3-b55ada6a4daa None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:26.673879 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83544234-4f1d-4e45-ba56-cceb1a8c2e73 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:26.697182 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:26.739358 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb9ffe3-3ba3-4b3f-8445-ed3b873283f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:26.739999 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb9ffe3-3ba3-4b3f-8445-ed3b873283f9 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:26.740272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb9ffe3-3ba3-4b3f-8445-ed3b873283f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:26.740561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb9ffe3-3ba3-4b3f-8445-ed3b873283f9 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:26.741773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb9ffe3-3ba3-4b3f-8445-ed3b873283f9 None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:26.743213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb9ffe3-3ba3-4b3f-8445-ed3b873283f9 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:26.745062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 178/350] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:26 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:26.745589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb9ffe3-3ba3-4b3f-8445-ed3b873283f9 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:26.757634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34bb01b-9bd3-478f-b8a3-b55ada6a4daa None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:26.814057 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4677012-b56c-43a1-a7a7-8aa0c7c4c50a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:26.814704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4677012-b56c-43a1-a7a7-8aa0c7c4c50a None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:26.814986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4677012-b56c-43a1-a7a7-8aa0c7c4c50a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:26.815260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4677012-b56c-43a1-a7a7-8aa0c7c4c50a None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:26.815934 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4677012-b56c-43a1-a7a7-8aa0c7c4c50a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:26.816236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4677012-b56c-43a1-a7a7-8aa0c7c4c50a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:26.818680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4677012-b56c-43a1-a7a7-8aa0c7c4c50a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:26.829905 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 173/351] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:26 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:26.830740 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4677012-b56c-43a1-a7a7-8aa0c7c4c50a None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:26.861810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:26.897318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb9ffe3-3ba3-4b3f-8445-ed3b873283f9 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:26.946604 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d159e01f-5c87-40a1-826f-f697dc854df3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:26.947201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d159e01f-5c87-40a1-826f-f697dc854df3 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:26.947482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d159e01f-5c87-40a1-826f-f697dc854df3 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:26.947756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d159e01f-5c87-40a1-826f-f697dc854df3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:26.948524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d159e01f-5c87-40a1-826f-f697dc854df3 None admin] RBAC: Adding query filter params (role.id=46cb171417764f9f9710877b8ec7becc, user.id=5a8c72fa028f4f479f600b9ac4bdeb3f, scope.system=all) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:26.948981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d159e01f-5c87-40a1-826f-f697dc854df3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:26.951155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d159e01f-5c87-40a1-826f-f697dc854df3 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:26.961241 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4677012-b56c-43a1-a7a7-8aa0c7c4c50a None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:26.966154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 179/352] 10.222.0.26 () {56 vars in 1308 bytes} [Wed Sep 28 12:01:26 2022] GET /identity/v3/role_assignments?user.id=5a8c72fa028f4f479f600b9ac4bdeb3f&scope.system=all&role.id=46cb171417764f9f9710877b8ec7becc => generated 510 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:26.967720 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2228b757-107b-4620-b4d3-5f6a0495aeb7 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:26.968257 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2228b757-107b-4620-b4d3-5f6a0495aeb7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:26.968777 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2228b757-107b-4620-b4d3-5f6a0495aeb7 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:26.970426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 180/353] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:26.971555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2228b757-107b-4620-b4d3-5f6a0495aeb7 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:27.002134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44b73bb-a5b0-4f25-b4eb-bc99809b2c3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:27.003466 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44b73bb-a5b0-4f25-b4eb-bc99809b2c3c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:27.003929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44b73bb-a5b0-4f25-b4eb-bc99809b2c3c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:27.004741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44b73bb-a5b0-4f25-b4eb-bc99809b2c3c None admin] PATH_INFO: `/v3/groups/66a838cf8186454db4bcf4d63b0f0f99` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:27.013502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44b73bb-a5b0-4f25-b4eb-bc99809b2c3c None admin] RBAC: Authorizing `identity:get_group(group_id=66a838cf8186454db4bcf4d63b0f0f99)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:27.016415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:27.017282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44b73bb-a5b0-4f25-b4eb-bc99809b2c3c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:27.019233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 174/354] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:26 2022] GET /identity/v3/groups/66a838cf8186454db4bcf4d63b0f0f99 => generated 226 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:27.025437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44b73bb-a5b0-4f25-b4eb-bc99809b2c3c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:27.109662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2228b757-107b-4620-b4d3-5f6a0495aeb7 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:27.150969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732cd0d6-4780-4674-974d-066c90e7b86c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:27.152013 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732cd0d6-4780-4674-974d-066c90e7b86c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:27.152536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732cd0d6-4780-4674-974d-066c90e7b86c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:27.153211 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:27.153676 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732cd0d6-4780-4674-974d-066c90e7b86c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:27.176030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44b73bb-a5b0-4f25-b4eb-bc99809b2c3c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:27.219003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb37d93-8d10-4c52-b0a7-f118687e24da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:27.219590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb37d93-8d10-4c52-b0a7-f118687e24da None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:27.219869 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb37d93-8d10-4c52-b0a7-f118687e24da None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:27.220140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb37d93-8d10-4c52-b0a7-f118687e24da None admin] PATH_INFO: `/v3/roles/member` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:27.234002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb37d93-8d10-4c52-b0a7-f118687e24da None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-8fb37d93-8d10-4c52-b0a7-f118687e24da None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:27.240198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:27.242037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:27.243382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:27.243382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:27.243382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:27.243382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:27.243382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:27.243382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:27.243382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:27.243382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:27.243382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:27.243382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:27.243382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:27.243382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:27.243382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 175/355] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:27 2022] GET /identity/v3/roles/member => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:27.243382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2f1c81-4474-4316-bb4a-28859628765c None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:27.244175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2f1c81-4474-4316-bb4a-28859628765c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:27.244175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2f1c81-4474-4316-bb4a-28859628765c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:27.266855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732cd0d6-4780-4674-974d-066c90e7b86c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:27.268613 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732cd0d6-4780-4674-974d-066c90e7b86c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:27.269223 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-7a2f1c81-4474-4316-bb4a-28859628765c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:27.270433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7a2f1c81-4474-4316-bb4a-28859628765c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:27.318749 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:27.366172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 176/356] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:27.367302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2f1c81-4474-4316-bb4a-28859628765c None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:27.395244 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 181/357] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:27.396679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732cd0d6-4780-4674-974d-066c90e7b86c None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:27.480184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:27.508995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2f1c81-4474-4316-bb4a-28859628765c None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:27.530565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732cd0d6-4780-4674-974d-066c90e7b86c None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:27.551060 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2ba587-f67f-4228-aa1a-be81742ed308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:27.553463 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ba587-f67f-4228-aa1a-be81742ed308 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:27.553751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ba587-f67f-4228-aa1a-be81742ed308 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:27.554029 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ba587-f67f-4228-aa1a-be81742ed308 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:27.554883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2ba587-f67f-4228-aa1a-be81742ed308 None admin] RBAC: Adding query filter params (group.id=66a838cf8186454db4bcf4d63b0f0f99, role.id=615e83fab2504b6298db3efaa7c15750, scope.project.id=47fa4785636148129d33fffc2096cd4b) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:27.555195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2ba587-f67f-4228-aa1a-be81742ed308 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:27.556677 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2ba587-f67f-4228-aa1a-be81742ed308 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:27.562890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 177/358] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Sep 28 12:01:27 2022] GET /identity/v3/role_assignments?group.id=66a838cf8186454db4bcf4d63b0f0f99&scope.project.id=47fa4785636148129d33fffc2096cd4b&role.id=615e83fab2504b6298db3efaa7c15750 => generated 260 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:27.563751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2ba587-f67f-4228-aa1a-be81742ed308 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:27.595145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7fc1af-7a6e-4aac-9be1-c655a68ab311 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:27.595773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7fc1af-7a6e-4aac-9be1-c655a68ab311 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:27.596050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7fc1af-7a6e-4aac-9be1-c655a68ab311 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:27.596343 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7fc1af-7a6e-4aac-9be1-c655a68ab311 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:27.597032 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7fc1af-7a6e-4aac-9be1-c655a68ab311 None admin] RBAC: Adding query filter params (name=member) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:27.597334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7fc1af-7a6e-4aac-9be1-c655a68ab311 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:27.599079 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7fc1af-7a6e-4aac-9be1-c655a68ab311 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:27.615168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 182/359] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:27.615826 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:27.627195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7fc1af-7a6e-4aac-9be1-c655a68ab311 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:27.745142 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2ba587-f67f-4228-aa1a-be81742ed308 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:27.757965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7fc1af-7a6e-4aac-9be1-c655a68ab311 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:27.785569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a4dbb2-ae13-47c5-92c3-aeed053a0537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:27.786757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a4dbb2-ae13-47c5-92c3-aeed053a0537 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:27.787044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a4dbb2-ae13-47c5-92c3-aeed053a0537 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:27.787321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a4dbb2-ae13-47c5-92c3-aeed053a0537 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:27.789122 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a4dbb2-ae13-47c5-92c3-aeed053a0537 None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:27.790159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a4dbb2-ae13-47c5-92c3-aeed053a0537 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:27.792572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 178/360] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:27 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:27.793984 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a4dbb2-ae13-47c5-92c3-aeed053a0537 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:27.808956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b50f81-1fdd-4fe9-9d5f-e8a019b59960 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:27.810262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:27.811097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b50f81-1fdd-4fe9-9d5f-e8a019b59960 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:27.811615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b50f81-1fdd-4fe9-9d5f-e8a019b59960 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:27.813015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b50f81-1fdd-4fe9-9d5f-e8a019b59960 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:27.898445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:27.902461 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b50f81-1fdd-4fe9-9d5f-e8a019b59960 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:27.904212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b50f81-1fdd-4fe9-9d5f-e8a019b59960 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:27.925759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a4dbb2-ae13-47c5-92c3-aeed053a0537 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:27.978596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a811f4-0808-4151-8eaa-cbb23510e1db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:27.979249 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a811f4-0808-4151-8eaa-cbb23510e1db None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:27.979535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a811f4-0808-4151-8eaa-cbb23510e1db None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:27.980147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a811f4-0808-4151-8eaa-cbb23510e1db None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:27.993225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a811f4-0808-4151-8eaa-cbb23510e1db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-92a811f4-0808-4151-8eaa-cbb23510e1db None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:27.998660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:27.999538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:28.000343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:28.000343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:28.000343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:28.000343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:28.000343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:28.000343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:28.000343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:28.000343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:28.000343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:28.000343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:28.000343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:28.000343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:28.000343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 179/361] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:01:27 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:28.001009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a811f4-0808-4151-8eaa-cbb23510e1db None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:28.030668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 183/362] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:28.032113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a748bd9-db41-48a9-be52-9d557986154f None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:28.032425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a748bd9-db41-48a9-be52-9d557986154f None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:28.032710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a748bd9-db41-48a9-be52-9d557986154f None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:28.034084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 184/363] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:28.035167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a748bd9-db41-48a9-be52-9d557986154f None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:28.098437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:28.139451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a811f4-0808-4151-8eaa-cbb23510e1db None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:28.153861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:28.161026 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a748bd9-db41-48a9-be52-9d557986154f None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:28.184408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf741889-7ac1-42ff-8b0a-6db68971c959 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:28.185054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf741889-7ac1-42ff-8b0a-6db68971c959 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:28.186609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf741889-7ac1-42ff-8b0a-6db68971c959 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:28.186609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf741889-7ac1-42ff-8b0a-6db68971c959 None admin] PATH_INFO: `/v3/groups/66a838cf8186454db4bcf4d63b0f0f99` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:28.186796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf741889-7ac1-42ff-8b0a-6db68971c959 None admin] RBAC: Authorizing `identity:get_group(group_id=66a838cf8186454db4bcf4d63b0f0f99)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:28.189194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf741889-7ac1-42ff-8b0a-6db68971c959 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:28.190925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 180/364] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:28 2022] GET /identity/v3/groups/66a838cf8186454db4bcf4d63b0f0f99 => generated 226 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:28.193208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf741889-7ac1-42ff-8b0a-6db68971c959 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:28.209935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6b0ea3-119c-4e59-a101-6033b3ffb43c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:28.210940 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6b0ea3-119c-4e59-a101-6033b3ffb43c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:28.211230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6b0ea3-119c-4e59-a101-6033b3ffb43c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:28.211502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6b0ea3-119c-4e59-a101-6033b3ffb43c None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:28.212467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6b0ea3-119c-4e59-a101-6033b3ffb43c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:28.212919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6b0ea3-119c-4e59-a101-6033b3ffb43c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:28.214594 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6b0ea3-119c-4e59-a101-6033b3ffb43c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:28.224344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 185/365] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:28 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:28.225698 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dedd84-72f2-49e7-b909-9dba787d6f89 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:28.225904 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dedd84-72f2-49e7-b909-9dba787d6f89 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:28.226184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dedd84-72f2-49e7-b909-9dba787d6f89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:28.250240 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-c5dedd84-72f2-49e7-b909-9dba787d6f89 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:28.251324 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c5dedd84-72f2-49e7-b909-9dba787d6f89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:28.327515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf741889-7ac1-42ff-8b0a-6db68971c959 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:28.334704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 186/366] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:28.335559 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dedd84-72f2-49e7-b909-9dba787d6f89 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:28.372399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46beb47e-810e-482d-bffe-8b18e78d7f50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:28.373141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46beb47e-810e-482d-bffe-8b18e78d7f50 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:28.373425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46beb47e-810e-482d-bffe-8b18e78d7f50 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:28.373701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46beb47e-810e-482d-bffe-8b18e78d7f50 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:28.374851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46beb47e-810e-482d-bffe-8b18e78d7f50 None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:28.376274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46beb47e-810e-482d-bffe-8b18e78d7f50 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:28.380564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 181/367] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:28 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:28.381485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46beb47e-810e-482d-bffe-8b18e78d7f50 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:28.392093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:28.467258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:28.484125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dedd84-72f2-49e7-b909-9dba787d6f89 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:28.520581 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46beb47e-810e-482d-bffe-8b18e78d7f50 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:28.529333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32dae9bb-9aa9-457f-b6dc-1dad5987e148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:28.529981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dae9bb-9aa9-457f-b6dc-1dad5987e148 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:28.530225 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dae9bb-9aa9-457f-b6dc-1dad5987e148 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:28.530497 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dae9bb-9aa9-457f-b6dc-1dad5987e148 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:28.531232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32dae9bb-9aa9-457f-b6dc-1dad5987e148 None admin] RBAC: Adding query filter params (group.id=66a838cf8186454db4bcf4d63b0f0f99, role.id=a3902a185ac144fb8e24f4e9feb039ed, scope.project.id=47fa4785636148129d33fffc2096cd4b) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:28.531540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32dae9bb-9aa9-457f-b6dc-1dad5987e148 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:28.533011 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32dae9bb-9aa9-457f-b6dc-1dad5987e148 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:28.540346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 187/368] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Sep 28 12:01:28 2022] GET /identity/v3/role_assignments?group.id=66a838cf8186454db4bcf4d63b0f0f99&scope.project.id=47fa4785636148129d33fffc2096cd4b&role.id=a3902a185ac144fb8e24f4e9feb039ed => generated 260 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:28.540346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32dae9bb-9aa9-457f-b6dc-1dad5987e148 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:28.602181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a43af25-7516-4bf0-9597-96a52cb828c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:28.602821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a43af25-7516-4bf0-9597-96a52cb828c1 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:28.603122 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a43af25-7516-4bf0-9597-96a52cb828c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:28.603377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a43af25-7516-4bf0-9597-96a52cb828c1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:28.683562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32dae9bb-9aa9-457f-b6dc-1dad5987e148 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:28.703780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a43af25-7516-4bf0-9597-96a52cb828c1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:28.705411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a43af25-7516-4bf0-9597-96a52cb828c1 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:28.716276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:28.736132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be18ba60-a05d-4ef0-b5da-1e2adcd8489e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:28.737399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be18ba60-a05d-4ef0-b5da-1e2adcd8489e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:28.737829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be18ba60-a05d-4ef0-b5da-1e2adcd8489e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:28.738247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be18ba60-a05d-4ef0-b5da-1e2adcd8489e None admin] PATH_INFO: `/v3/groups/66a838cf8186454db4bcf4d63b0f0f99` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:28.740964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be18ba60-a05d-4ef0-b5da-1e2adcd8489e None admin] RBAC: Authorizing `identity:get_group(group_id=66a838cf8186454db4bcf4d63b0f0f99)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:28.743146 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be18ba60-a05d-4ef0-b5da-1e2adcd8489e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:28.746445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 188/369] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:28 2022] GET /identity/v3/groups/66a838cf8186454db4bcf4d63b0f0f99 => generated 226 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:28.747445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be18ba60-a05d-4ef0-b5da-1e2adcd8489e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:28.782261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:28.833970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 182/370] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:28.834954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a43af25-7516-4bf0-9597-96a52cb828c1 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:28.873731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be18ba60-a05d-4ef0-b5da-1e2adcd8489e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:28.910855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef54ae0-1b9c-4f41-891a-d9df29a6770e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:28.912052 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef54ae0-1b9c-4f41-891a-d9df29a6770e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:28.912373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef54ae0-1b9c-4f41-891a-d9df29a6770e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:28.912649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef54ae0-1b9c-4f41-891a-d9df29a6770e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:28.924537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef54ae0-1b9c-4f41-891a-d9df29a6770e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-3ef54ae0-1b9c-4f41-891a-d9df29a6770e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:28.930400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:28.931258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:28.932033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:28.932033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:28.932033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:28.932033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:28.932033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:28.932033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:28.932033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:28.932033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:28.932033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:28.932033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:28.932033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:01:28.932033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:28.932033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 189/371] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:01:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:28.936741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef54ae0-1b9c-4f41-891a-d9df29a6770e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:28.942640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a43af25-7516-4bf0-9597-96a52cb828c1 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:28.956686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:28.984924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0df232-d8d0-4e82-8105-9f976ac3e941 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:28.985643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0df232-d8d0-4e82-8105-9f976ac3e941 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:28.985930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0df232-d8d0-4e82-8105-9f976ac3e941 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:28.986208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0df232-d8d0-4e82-8105-9f976ac3e941 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:28.987108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0df232-d8d0-4e82-8105-9f976ac3e941 None admin] RBAC: Adding query filter params (group.id=66a838cf8186454db4bcf4d63b0f0f99, role.id=615e83fab2504b6298db3efaa7c15750, scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:28.987429 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0df232-d8d0-4e82-8105-9f976ac3e941 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:28.989178 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0df232-d8d0-4e82-8105-9f976ac3e941 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:28.996402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 183/372] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Sep 28 12:01:28 2022] GET /identity/v3/role_assignments?group.id=66a838cf8186454db4bcf4d63b0f0f99&scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4&role.id=615e83fab2504b6298db3efaa7c15750 => generated 260 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:28.997324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0df232-d8d0-4e82-8105-9f976ac3e941 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:29.065790 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:29.093050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef54ae0-1b9c-4f41-891a-d9df29a6770e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:29.134679 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0df232-d8d0-4e82-8105-9f976ac3e941 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:29.154729 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2328fc-2734-41a7-9b51-40f5313c5146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:29.155383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2328fc-2734-41a7-9b51-40f5313c5146 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:29.155648 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2328fc-2734-41a7-9b51-40f5313c5146 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:29.155921 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2328fc-2734-41a7-9b51-40f5313c5146 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:29.161233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2328fc-2734-41a7-9b51-40f5313c5146 None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:29.162717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2328fc-2734-41a7-9b51-40f5313c5146 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:29.164661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 190/373] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:28 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:29.170224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2328fc-2734-41a7-9b51-40f5313c5146 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:29.172929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba11ca6e-3852-46c4-bf1e-761790336b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:29.173656 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba11ca6e-3852-46c4-bf1e-761790336b6f None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:29.174039 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba11ca6e-3852-46c4-bf1e-761790336b6f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:29.174329 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba11ca6e-3852-46c4-bf1e-761790336b6f None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:29.175017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba11ca6e-3852-46c4-bf1e-761790336b6f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:29.175325 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba11ca6e-3852-46c4-bf1e-761790336b6f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:29.176762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba11ca6e-3852-46c4-bf1e-761790336b6f None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:29.187651 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 184/374] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:29.197573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba11ca6e-3852-46c4-bf1e-761790336b6f None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:29.261893 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:29.318722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2328fc-2734-41a7-9b51-40f5313c5146 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:29.361306 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c81d755-1309-496c-9b2d-a8ac1d664029 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:29.362000 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c81d755-1309-496c-9b2d-a8ac1d664029 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:29.362537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c81d755-1309-496c-9b2d-a8ac1d664029 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:29.362819 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c81d755-1309-496c-9b2d-a8ac1d664029 None admin] PATH_INFO: `/v3/groups/66a838cf8186454db4bcf4d63b0f0f99` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:29.364077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c81d755-1309-496c-9b2d-a8ac1d664029 None admin] RBAC: Authorizing `identity:get_group(group_id=66a838cf8186454db4bcf4d63b0f0f99)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:29.366912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c81d755-1309-496c-9b2d-a8ac1d664029 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:29.369065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 191/375] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:29 2022] GET /identity/v3/groups/66a838cf8186454db4bcf4d63b0f0f99 => generated 226 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:29.374692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c81d755-1309-496c-9b2d-a8ac1d664029 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:29.400786 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:29.406746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba11ca6e-3852-46c4-bf1e-761790336b6f None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:29.451728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e665694c-c6f6-4534-8852-4d482ad558ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:29.452430 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e665694c-c6f6-4534-8852-4d482ad558ac None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:29.452751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e665694c-c6f6-4534-8852-4d482ad558ac None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:29.453068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e665694c-c6f6-4534-8852-4d482ad558ac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:29.516024 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c81d755-1309-496c-9b2d-a8ac1d664029 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:29.537972 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e665694c-c6f6-4534-8852-4d482ad558ac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:29.539915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e665694c-c6f6-4534-8852-4d482ad558ac None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:29.577546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:29.595754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf1125e-42f8-4310-98b7-e5c2b941d024 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:29.596730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf1125e-42f8-4310-98b7-e5c2b941d024 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:29.597255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf1125e-42f8-4310-98b7-e5c2b941d024 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:29.597718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf1125e-42f8-4310-98b7-e5c2b941d024 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:29.598674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf1125e-42f8-4310-98b7-e5c2b941d024 None admin] RBAC: Adding query filter params (group.id=66a838cf8186454db4bcf4d63b0f0f99, role.id=a3902a185ac144fb8e24f4e9feb039ed, scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:29.599183 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf1125e-42f8-4310-98b7-e5c2b941d024 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:29.604979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf1125e-42f8-4310-98b7-e5c2b941d024 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:29.615064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 192/376] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Sep 28 12:01:29 2022] GET /identity/v3/role_assignments?group.id=66a838cf8186454db4bcf4d63b0f0f99&scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4&role.id=a3902a185ac144fb8e24f4e9feb039ed => generated 260 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:29.658623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 185/377] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:29.664507 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf1125e-42f8-4310-98b7-e5c2b941d024 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:29.773708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:29.794381 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf1125e-42f8-4310-98b7-e5c2b941d024 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:29.832274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffef121-e877-4090-a9e3-e47fa86a3bdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:29.832863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffef121-e877-4090-a9e3-e47fa86a3bdd None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:29.833195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffef121-e877-4090-a9e3-e47fa86a3bdd None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:29.833464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffef121-e877-4090-a9e3-e47fa86a3bdd None admin] PATH_INFO: `/v3/projects/8d6b5c1f82154555b8d48a213cdda1ed` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:29.834610 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffef121-e877-4090-a9e3-e47fa86a3bdd None admin] RBAC: Authorizing `identity:get_project(project_id=8d6b5c1f82154555b8d48a213cdda1ed)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:29.836124 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffef121-e877-4090-a9e3-e47fa86a3bdd None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:29.838485 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 193/378] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:29 2022] GET /identity/v3/projects/8d6b5c1f82154555b8d48a213cdda1ed => generated 344 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:29.844042 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e665694c-c6f6-4534-8852-4d482ad558ac None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:29.960386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e665694c-c6f6-4534-8852-4d482ad558ac None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:29.992605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:30.003679 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94cbd4f-825f-49a4-8774-24bdb75a5843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:30.004331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94cbd4f-825f-49a4-8774-24bdb75a5843 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:30.004619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94cbd4f-825f-49a4-8774-24bdb75a5843 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:30.004929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94cbd4f-825f-49a4-8774-24bdb75a5843 None admin] PATH_INFO: `/v3/groups/0cb97143c35a4b5baea9d1bb8ef06f53` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:30.009983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94cbd4f-825f-49a4-8774-24bdb75a5843 None admin] RBAC: Authorizing `identity:get_group(group_id=0cb97143c35a4b5baea9d1bb8ef06f53)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:30.011426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94cbd4f-825f-49a4-8774-24bdb75a5843 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:30.013390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 186/379] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:29 2022] GET /identity/v3/groups/0cb97143c35a4b5baea9d1bb8ef06f53 => generated 229 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:30.019276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffef121-e877-4090-a9e3-e47fa86a3bdd None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:30.155590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffef121-e877-4090-a9e3-e47fa86a3bdd None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:30.202123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6004d71-0d0d-482d-91f7-ed51d5828201 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:30.203094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6004d71-0d0d-482d-91f7-ed51d5828201 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:30.203574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6004d71-0d0d-482d-91f7-ed51d5828201 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:30.204046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6004d71-0d0d-482d-91f7-ed51d5828201 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:30.205010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6004d71-0d0d-482d-91f7-ed51d5828201 None admin] RBAC: Adding query filter params (group.id=0cb97143c35a4b5baea9d1bb8ef06f53, role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=8d6b5c1f82154555b8d48a213cdda1ed) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:30.205524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6004d71-0d0d-482d-91f7-ed51d5828201 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:30.207182 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6004d71-0d0d-482d-91f7-ed51d5828201 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:30.216774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 194/380] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Sep 28 12:01:30 2022] GET /identity/v3/role_assignments?group.id=0cb97143c35a4b5baea9d1bb8ef06f53&scope.project.id=8d6b5c1f82154555b8d48a213cdda1ed&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 260 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:30.286987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79511e7-c39b-4a9b-9bc8-52370bc46099 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:30.287543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79511e7-c39b-4a9b-9bc8-52370bc46099 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:30.287543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79511e7-c39b-4a9b-9bc8-52370bc46099 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:30.289190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 187/381] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:30.301140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fe242c-a46d-46d2-8909-e5d43fd6bd81 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:30.301471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fe242c-a46d-46d2-8909-e5d43fd6bd81 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:30.301737 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fe242c-a46d-46d2-8909-e5d43fd6bd81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:30.328062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-86fe242c-a46d-46d2-8909-e5d43fd6bd81 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:30.329298 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-86fe242c-a46d-46d2-8909-e5d43fd6bd81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:30.342571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:30.414004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 195/382] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:30.978151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79511e7-c39b-4a9b-9bc8-52370bc46099 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:31.101971 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79511e7-c39b-4a9b-9bc8-52370bc46099 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:31.143726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28dbd61-96a6-472a-9b24-480490086e3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:31.144394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28dbd61-96a6-472a-9b24-480490086e3a None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:31.144678 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28dbd61-96a6-472a-9b24-480490086e3a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:31.144956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28dbd61-96a6-472a-9b24-480490086e3a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:31.155942 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28dbd61-96a6-472a-9b24-480490086e3a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-a28dbd61-96a6-472a-9b24-480490086e3a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:31.164898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:31.165815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:31.166637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:31.166637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:31.166637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:31.166637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:31.166637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:31.166637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:31.166637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:31.166637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:31.166637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:31.166637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:31.166637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:31.166637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:31.169345 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 188/383] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:31.174922 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fe242c-a46d-46d2-8909-e5d43fd6bd81 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:31.315079 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fe242c-a46d-46d2-8909-e5d43fd6bd81 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:31.369055 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902b6178-ab97-4245-89cf-b8406bc1375e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:31.369686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902b6178-ab97-4245-89cf-b8406bc1375e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:31.369972 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902b6178-ab97-4245-89cf-b8406bc1375e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:31.370251 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902b6178-ab97-4245-89cf-b8406bc1375e None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:31.370940 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902b6178-ab97-4245-89cf-b8406bc1375e None admin] RBAC: Adding query filter params (name=member) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:31.371247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902b6178-ab97-4245-89cf-b8406bc1375e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:31.372668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902b6178-ab97-4245-89cf-b8406bc1375e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:31.381943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 196/384] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:31.387250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28dbd61-96a6-472a-9b24-480490086e3a None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:31.472488 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:31.542392 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28dbd61-96a6-472a-9b24-480490086e3a None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:31.588362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e250a752-facd-4eff-a620-9404034b53af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:31.588978 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e250a752-facd-4eff-a620-9404034b53af None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:31.589304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e250a752-facd-4eff-a620-9404034b53af None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:31.589580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e250a752-facd-4eff-a620-9404034b53af None admin] PATH_INFO: `/v3/groups/66a838cf8186454db4bcf4d63b0f0f99` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:31.590740 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e250a752-facd-4eff-a620-9404034b53af None admin] RBAC: Authorizing `identity:get_group(group_id=66a838cf8186454db4bcf4d63b0f0f99)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:31.592181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e250a752-facd-4eff-a620-9404034b53af None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:31.594135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 189/385] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:31 2022] GET /identity/v3/groups/66a838cf8186454db4bcf4d63b0f0f99 => generated 226 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:31.601243 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902b6178-ab97-4245-89cf-b8406bc1375e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:31.607697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e8f2b9-7152-441e-a1e5-13cc63cd2645 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:31.607984 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e8f2b9-7152-441e-a1e5-13cc63cd2645 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:31.608264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e8f2b9-7152-441e-a1e5-13cc63cd2645 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:31.610136 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 190/386] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:01:31.618768 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:31.621730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb7819b-a20a-4c2d-aecc-f4e8b193ff78 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:31.621935 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb7819b-a20a-4c2d-aecc-f4e8b193ff78 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:31.622209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb7819b-a20a-4c2d-aecc-f4e8b193ff78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:31.646571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-2bb7819b-a20a-4c2d-aecc-f4e8b193ff78 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:31.648120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2bb7819b-a20a-4c2d-aecc-f4e8b193ff78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:31.736849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 191/387] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:31.789849 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902b6178-ab97-4245-89cf-b8406bc1375e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:31.838934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dd7a23-4063-4e5c-bb2b-d4372ade60b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:31.839484 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dd7a23-4063-4e5c-bb2b-d4372ade60b3 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:31.839767 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dd7a23-4063-4e5c-bb2b-d4372ade60b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:31.840043 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dd7a23-4063-4e5c-bb2b-d4372ade60b3 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:31.841231 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93dd7a23-4063-4e5c-bb2b-d4372ade60b3 None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:31.842683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93dd7a23-4063-4e5c-bb2b-d4372ade60b3 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:31.849191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 197/388] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:31 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:31.850707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb7819b-a20a-4c2d-aecc-f4e8b193ff78 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:31.918692 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:31.970600 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb7819b-a20a-4c2d-aecc-f4e8b193ff78 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:32.015125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca66d86-6fcc-41d9-888c-f5cfbe25fcfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:32.015790 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca66d86-6fcc-41d9-888c-f5cfbe25fcfe None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:32.017179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca66d86-6fcc-41d9-888c-f5cfbe25fcfe None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:32.017462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca66d86-6fcc-41d9-888c-f5cfbe25fcfe None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b/groups/66a838cf8186454db4bcf4d63b0f0f99/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:32.019470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca66d86-6fcc-41d9-888c-f5cfbe25fcfe None admin] RBAC: Authorizing `identity:create_grant(project_id=47fa4785636148129d33fffc2096cd4b, group_id=66a838cf8186454db4bcf4d63b0f0f99, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:32.020963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca66d86-6fcc-41d9-888c-f5cfbe25fcfe None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:32.040525 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 192/389] 10.222.0.26 () {58 vars in 1442 bytes} [Wed Sep 28 12:01:31 2022] PUT /identity/v3/projects/47fa4785636148129d33fffc2096cd4b/groups/66a838cf8186454db4bcf4d63b0f0f99/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:01:32.084172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609b3b08-b77e-4abd-ada7-b7f5d281a6a0 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:32.084846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609b3b08-b77e-4abd-ada7-b7f5d281a6a0 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:32.085352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609b3b08-b77e-4abd-ada7-b7f5d281a6a0 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:32.087537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 198/390] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:32.092995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de8af8e-7eed-4905-b0eb-371eb1c594e0 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:32.093248 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de8af8e-7eed-4905-b0eb-371eb1c594e0 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:32.093485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de8af8e-7eed-4905-b0eb-371eb1c594e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:32.109257 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609b3b08-b77e-4abd-ada7-b7f5d281a6a0 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:32.118836 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-1de8af8e-7eed-4905-b0eb-371eb1c594e0 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:32.123551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1de8af8e-7eed-4905-b0eb-371eb1c594e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:32.250763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 193/391] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:32.259467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:32.296067 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609b3b08-b77e-4abd-ada7-b7f5d281a6a0 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:32.337261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49b7d86-b7ec-4c59-9034-a3c82949ff28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:32.337910 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49b7d86-b7ec-4c59-9034-a3c82949ff28 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:32.338185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49b7d86-b7ec-4c59-9034-a3c82949ff28 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:32.338450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49b7d86-b7ec-4c59-9034-a3c82949ff28 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:32.361239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49b7d86-b7ec-4c59-9034-a3c82949ff28 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-d49b7d86-b7ec-4c59-9034-a3c82949ff28 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:32.366841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:32.367774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:32.368597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:32.368597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:32.368597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:32.368597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:32.368597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:32.368597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:32.368597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:32.368597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:32.368597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:32.368597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:32.368597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:32.368597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:32.368597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 199/392] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:01:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:32.374983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de8af8e-7eed-4905-b0eb-371eb1c594e0 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:32.435002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:32.479371 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93011cb8-9963-48ca-81a4-eae9d8230995 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:32.479697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93011cb8-9963-48ca-81a4-eae9d8230995 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:32.479909 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93011cb8-9963-48ca-81a4-eae9d8230995 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:32.488279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 200/393] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:32.501312 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2f9f7f-3614-4457-8e6f-09c4b38d3c62 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:32.501596 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2f9f7f-3614-4457-8e6f-09c4b38d3c62 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:32.501867 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2f9f7f-3614-4457-8e6f-09c4b38d3c62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:32.507471 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de8af8e-7eed-4905-b0eb-371eb1c594e0 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:32.536264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-2b2f9f7f-3614-4457-8e6f-09c4b38d3c62 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:32.545398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2b2f9f7f-3614-4457-8e6f-09c4b38d3c62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:32.553102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562fb8b6-a80d-4c18-8427-25a670cb5181 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:32.553749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562fb8b6-a80d-4c18-8427-25a670cb5181 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:32.554023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562fb8b6-a80d-4c18-8427-25a670cb5181 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:32.554288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562fb8b6-a80d-4c18-8427-25a670cb5181 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:32.554962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562fb8b6-a80d-4c18-8427-25a670cb5181 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:32.555261 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562fb8b6-a80d-4c18-8427-25a670cb5181 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:32.556654 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562fb8b6-a80d-4c18-8427-25a670cb5181 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:32.569498 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 194/394] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:32.584410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562fb8b6-a80d-4c18-8427-25a670cb5181 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:32.662177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 201/395] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:32.663105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2f9f7f-3614-4457-8e6f-09c4b38d3c62 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:32.732094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:32.775417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2f9f7f-3614-4457-8e6f-09c4b38d3c62 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:32.794504 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562fb8b6-a80d-4c18-8427-25a670cb5181 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:32.812340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:32.815599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ba4c45-5153-4815-a745-2795533245cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:32.816569 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ba4c45-5153-4815-a745-2795533245cb None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:32.816846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ba4c45-5153-4815-a745-2795533245cb None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:32.817115 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ba4c45-5153-4815-a745-2795533245cb None admin] PATH_INFO: `/v3/roles/member` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:32.832471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ba4c45-5153-4815-a745-2795533245cb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-79ba4c45-5153-4815-a745-2795533245cb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:32.842424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:32.843320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:32.844099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:32.844099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:32.844099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:32.844099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:32.844099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:32.844099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:32.844099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:32.844099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:32.844099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:32.844099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:32.844099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:32.844099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:32.845416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 202/396] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:32.857825 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ba4c45-5153-4815-a745-2795533245cb None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:32.870568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1951c8-71a3-48c0-ab8a-33fa3ac5a5f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:32.871259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1951c8-71a3-48c0-ab8a-33fa3ac5a5f8 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:32.871524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1951c8-71a3-48c0-ab8a-33fa3ac5a5f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:32.871800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1951c8-71a3-48c0-ab8a-33fa3ac5a5f8 None admin] PATH_INFO: `/v3/groups/66a838cf8186454db4bcf4d63b0f0f99` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:32.874951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1951c8-71a3-48c0-ab8a-33fa3ac5a5f8 None admin] RBAC: Authorizing `identity:get_group(group_id=66a838cf8186454db4bcf4d63b0f0f99)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:32.877011 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1951c8-71a3-48c0-ab8a-33fa3ac5a5f8 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:32.882805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 195/397] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:32 2022] GET /identity/v3/groups/66a838cf8186454db4bcf4d63b0f0f99 => generated 226 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:32.890666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1951c8-71a3-48c0-ab8a-33fa3ac5a5f8 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:33.007890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1951c8-71a3-48c0-ab8a-33fa3ac5a5f8 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:33.049686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685b8872-c29d-4ede-b42c-411c24118205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:33.050470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685b8872-c29d-4ede-b42c-411c24118205 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:33.050748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685b8872-c29d-4ede-b42c-411c24118205 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:33.051021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685b8872-c29d-4ede-b42c-411c24118205 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:33.052787 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685b8872-c29d-4ede-b42c-411c24118205 None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:33.054363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685b8872-c29d-4ede-b42c-411c24118205 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:33.057839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ba4c45-5153-4815-a745-2795533245cb None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:33.060379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 196/398] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:32 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:33.061431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685b8872-c29d-4ede-b42c-411c24118205 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:33.091846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:33.108669 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6883fbc-3823-49bd-aa24-fbdd28ca1b21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:33.109646 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6883fbc-3823-49bd-aa24-fbdd28ca1b21 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:33.110201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6883fbc-3823-49bd-aa24-fbdd28ca1b21 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:33.110685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6883fbc-3823-49bd-aa24-fbdd28ca1b21 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:33.111556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6883fbc-3823-49bd-aa24-fbdd28ca1b21 None admin] RBAC: Adding query filter params (name=member) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:33.112060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6883fbc-3823-49bd-aa24-fbdd28ca1b21 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:33.114213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6883fbc-3823-49bd-aa24-fbdd28ca1b21 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:33.116666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:33.131773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 203/399] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:33.133025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6883fbc-3823-49bd-aa24-fbdd28ca1b21 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:33.202733 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685b8872-c29d-4ede-b42c-411c24118205 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:33.245652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e3119d-0602-4829-bbb4-e66ee18b81bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:33.246478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e3119d-0602-4829-bbb4-e66ee18b81bc None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:33.246851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e3119d-0602-4829-bbb4-e66ee18b81bc None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:33.247124 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e3119d-0602-4829-bbb4-e66ee18b81bc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:33.259788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e3119d-0602-4829-bbb4-e66ee18b81bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-05e3119d-0602-4829-bbb4-e66ee18b81bc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:33.266739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:33.267724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:33.268515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:33.268515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:33.268515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:33.268515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:33.268515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:33.268515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:33.268515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:33.268515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:33.268515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:33.268515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:33.268515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:33.268515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:33.268515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 197/400] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:01:33 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:33.269176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e3119d-0602-4829-bbb4-e66ee18b81bc None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:33.281120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6883fbc-3823-49bd-aa24-fbdd28ca1b21 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:33.321015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd09c5f-629b-42b8-9c94-cc0844e04017 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:33.322336 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd09c5f-629b-42b8-9c94-cc0844e04017 None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:33.322336 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd09c5f-629b-42b8-9c94-cc0844e04017 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:33.322336 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd09c5f-629b-42b8-9c94-cc0844e04017 None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b/groups/66a838cf8186454db4bcf4d63b0f0f99/roles/a3902a185ac144fb8e24f4e9feb039ed` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:33.324397 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd09c5f-629b-42b8-9c94-cc0844e04017 None admin] RBAC: Authorizing `identity:create_grant(project_id=47fa4785636148129d33fffc2096cd4b, group_id=66a838cf8186454db4bcf4d63b0f0f99, role_id=a3902a185ac144fb8e24f4e9feb039ed)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:33.325842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd09c5f-629b-42b8-9c94-cc0844e04017 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:33.344377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 204/401] 10.222.0.26 () {58 vars in 1442 bytes} [Wed Sep 28 12:01:33 2022] PUT /identity/v3/projects/47fa4785636148129d33fffc2096cd4b/groups/66a838cf8186454db4bcf4d63b0f0f99/roles/a3902a185ac144fb8e24f4e9feb039ed => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:01:33.344377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4780c06-de4d-4ec7-b189-9de820afc6d6 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:33.344377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4780c06-de4d-4ec7-b189-9de820afc6d6 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:33.344377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4780c06-de4d-4ec7-b189-9de820afc6d6 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:33.344377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 205/402] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:33 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:33.344377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4780c06-de4d-4ec7-b189-9de820afc6d6 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:33.404518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:33.416875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e3119d-0602-4829-bbb4-e66ee18b81bc None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:33.419051 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:33.456394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59d66c9-d014-49f8-a6da-17efab55e36a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:33.458429 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59d66c9-d014-49f8-a6da-17efab55e36a None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:33.459014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59d66c9-d014-49f8-a6da-17efab55e36a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:33.459472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59d66c9-d014-49f8-a6da-17efab55e36a None admin] PATH_INFO: `/v3/groups/66a838cf8186454db4bcf4d63b0f0f99` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:33.460931 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59d66c9-d014-49f8-a6da-17efab55e36a None admin] RBAC: Authorizing `identity:get_group(group_id=66a838cf8186454db4bcf4d63b0f0f99)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:33.462956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59d66c9-d014-49f8-a6da-17efab55e36a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:33.465317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 198/403] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:33 2022] GET /identity/v3/groups/66a838cf8186454db4bcf4d63b0f0f99 => generated 226 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:33.466819 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f069e9-b779-4550-bcbe-4802c3391c22 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:33.467282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f069e9-b779-4550-bcbe-4802c3391c22 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:33.467720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f069e9-b779-4550-bcbe-4802c3391c22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:33.482757 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4780c06-de4d-4ec7-b189-9de820afc6d6 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:33.496472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-01f069e9-b779-4550-bcbe-4802c3391c22 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:33.497706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-01f069e9-b779-4550-bcbe-4802c3391c22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:33.541978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee667709-d2f6-4af2-8f9c-2fc8d1fae7c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:33.543607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee667709-d2f6-4af2-8f9c-2fc8d1fae7c7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:33.543607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee667709-d2f6-4af2-8f9c-2fc8d1fae7c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:33.543607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee667709-d2f6-4af2-8f9c-2fc8d1fae7c7 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:33.543862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee667709-d2f6-4af2-8f9c-2fc8d1fae7c7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:33.544147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee667709-d2f6-4af2-8f9c-2fc8d1fae7c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:33.545489 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee667709-d2f6-4af2-8f9c-2fc8d1fae7c7 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:33.557083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 206/404] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:33 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:33.558075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee667709-d2f6-4af2-8f9c-2fc8d1fae7c7 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:33.590133 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 199/405] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:33.591197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f069e9-b779-4550-bcbe-4802c3391c22 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:33.677855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee667709-d2f6-4af2-8f9c-2fc8d1fae7c7 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:33.694049 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:33.726563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d222eff0-362f-4d7e-aea6-f8667266ddda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:33.727534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d222eff0-362f-4d7e-aea6-f8667266ddda None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:33.728001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d222eff0-362f-4d7e-aea6-f8667266ddda None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:33.728481 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d222eff0-362f-4d7e-aea6-f8667266ddda None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:33.730164 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d222eff0-362f-4d7e-aea6-f8667266ddda None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:33.731810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d222eff0-362f-4d7e-aea6-f8667266ddda None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:33.734525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 207/406] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:33 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:33.743448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d222eff0-362f-4d7e-aea6-f8667266ddda None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:33.749425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:33.755604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f069e9-b779-4550-bcbe-4802c3391c22 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:33.812422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc78ed8f-4c4b-4c82-b952-6e88d445892d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:33.812978 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc78ed8f-4c4b-4c82-b952-6e88d445892d None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:33.813489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc78ed8f-4c4b-4c82-b952-6e88d445892d None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:33.813759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc78ed8f-4c4b-4c82-b952-6e88d445892d None admin] PATH_INFO: `/v3/groups/66a838cf8186454db4bcf4d63b0f0f99` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:33.815680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc78ed8f-4c4b-4c82-b952-6e88d445892d None admin] RBAC: Authorizing `identity:get_group(group_id=66a838cf8186454db4bcf4d63b0f0f99)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:33.817140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc78ed8f-4c4b-4c82-b952-6e88d445892d None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:33.819455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 200/407] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:33 2022] GET /identity/v3/groups/66a838cf8186454db4bcf4d63b0f0f99 => generated 226 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:33.824937 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc78ed8f-4c4b-4c82-b952-6e88d445892d None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:33.949943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d222eff0-362f-4d7e-aea6-f8667266ddda None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:33.990557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc78ed8f-4c4b-4c82-b952-6e88d445892d None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:33.990870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc462c45-0377-4e7d-9e56-be7228d24ca9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:33.992408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc462c45-0377-4e7d-9e56-be7228d24ca9 None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:33.992408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc462c45-0377-4e7d-9e56-be7228d24ca9 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:33.992408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc462c45-0377-4e7d-9e56-be7228d24ca9 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4/groups/66a838cf8186454db4bcf4d63b0f0f99/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:33.994288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc462c45-0377-4e7d-9e56-be7228d24ca9 None admin] RBAC: Authorizing `identity:create_grant(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4, group_id=66a838cf8186454db4bcf4d63b0f0f99, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:33.996017 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc462c45-0377-4e7d-9e56-be7228d24ca9 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:34.021027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 208/408] 10.222.0.26 () {58 vars in 1442 bytes} [Wed Sep 28 12:01:33 2022] PUT /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4/groups/66a838cf8186454db4bcf4d63b0f0f99/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:01:34.022004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc462c45-0377-4e7d-9e56-be7228d24ca9 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:34.044739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6684c261-3694-471b-b3cf-bac8019358f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:34.045900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6684c261-3694-471b-b3cf-bac8019358f7 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:34.046390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6684c261-3694-471b-b3cf-bac8019358f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:34.046889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6684c261-3694-471b-b3cf-bac8019358f7 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:34.050170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6684c261-3694-471b-b3cf-bac8019358f7 None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:34.052000 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6684c261-3694-471b-b3cf-bac8019358f7 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:34.056198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 201/409] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:33 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:34.062956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6684c261-3694-471b-b3cf-bac8019358f7 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:34.092978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:34.146079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:34.225151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc462c45-0377-4e7d-9e56-be7228d24ca9 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:34.273928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6684c261-3694-471b-b3cf-bac8019358f7 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:34.280733 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c40d0b-ed93-4b9f-87e7-7354a5bf0263 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:34.281298 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c40d0b-ed93-4b9f-87e7-7354a5bf0263 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:34.281834 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c40d0b-ed93-4b9f-87e7-7354a5bf0263 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:34.282005 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c40d0b-ed93-4b9f-87e7-7354a5bf0263 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:34.296236 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c40d0b-ed93-4b9f-87e7-7354a5bf0263 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-67c40d0b-ed93-4b9f-87e7-7354a5bf0263 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:34.302478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:34.303410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:34.304173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:34.304173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:34.304173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:34.304173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:34.304173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:34.304173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:34.304173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:34.304173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:34.304173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:34.304173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:34.304173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:01:34.304173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:34.311197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 209/410] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:01:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:34.311197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c40d0b-ed93-4b9f-87e7-7354a5bf0263 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:34.329650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816eb2cd-cb2a-4f28-b1ba-ee16dab74741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:34.330616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816eb2cd-cb2a-4f28-b1ba-ee16dab74741 None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:34.331338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816eb2cd-cb2a-4f28-b1ba-ee16dab74741 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:34.332254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816eb2cd-cb2a-4f28-b1ba-ee16dab74741 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4/groups/66a838cf8186454db4bcf4d63b0f0f99/roles/a3902a185ac144fb8e24f4e9feb039ed` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:34.335770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816eb2cd-cb2a-4f28-b1ba-ee16dab74741 None admin] RBAC: Authorizing `identity:create_grant(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4, group_id=66a838cf8186454db4bcf4d63b0f0f99, role_id=a3902a185ac144fb8e24f4e9feb039ed)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:34.337999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816eb2cd-cb2a-4f28-b1ba-ee16dab74741 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:34.363620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 202/411] 10.222.0.26 () {58 vars in 1442 bytes} [Wed Sep 28 12:01:34 2022] PUT /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4/groups/66a838cf8186454db4bcf4d63b0f0f99/roles/a3902a185ac144fb8e24f4e9feb039ed => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:01:34.445048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:34.451955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c40d0b-ed93-4b9f-87e7-7354a5bf0263 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:34.496339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad92867-2a5a-4281-9d4c-bc9f517cd749 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:34.496339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad92867-2a5a-4281-9d4c-bc9f517cd749 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:34.496339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad92867-2a5a-4281-9d4c-bc9f517cd749 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:34.496339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad92867-2a5a-4281-9d4c-bc9f517cd749 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:34.496692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad92867-2a5a-4281-9d4c-bc9f517cd749 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:34.496692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad92867-2a5a-4281-9d4c-bc9f517cd749 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:34.497672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad92867-2a5a-4281-9d4c-bc9f517cd749 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:34.516623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 210/412] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:34.522521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816eb2cd-cb2a-4f28-b1ba-ee16dab74741 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:34.666143 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:34.672796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816eb2cd-cb2a-4f28-b1ba-ee16dab74741 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:34.719205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29fc554-620b-4be4-8b9d-56d485e17c62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:34.719909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29fc554-620b-4be4-8b9d-56d485e17c62 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:34.720189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29fc554-620b-4be4-8b9d-56d485e17c62 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:34.720513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29fc554-620b-4be4-8b9d-56d485e17c62 None admin] PATH_INFO: `/v3/groups/0cb97143c35a4b5baea9d1bb8ef06f53` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:34.721649 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29fc554-620b-4be4-8b9d-56d485e17c62 None admin] RBAC: Authorizing `identity:get_group(group_id=0cb97143c35a4b5baea9d1bb8ef06f53)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:34.723376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29fc554-620b-4be4-8b9d-56d485e17c62 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:34.725745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 203/413] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:34 2022] GET /identity/v3/groups/0cb97143c35a4b5baea9d1bb8ef06f53 => generated 229 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:34.731963 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad92867-2a5a-4281-9d4c-bc9f517cd749 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:34.797729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1590b0fe-c46f-435f-a5db-4f203daefa2c None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:34.797993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1590b0fe-c46f-435f-a5db-4f203daefa2c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:34.798282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1590b0fe-c46f-435f-a5db-4f203daefa2c None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:34.799620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 204/414] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:01:34.809995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e376ac-0070-414b-9a9a-bbe2f0f95bc2 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:34.810334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e376ac-0070-414b-9a9a-bbe2f0f95bc2 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:34.810593 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e376ac-0070-414b-9a9a-bbe2f0f95bc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:34.830548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-51e376ac-0070-414b-9a9a-bbe2f0f95bc2 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:34.831721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-51e376ac-0070-414b-9a9a-bbe2f0f95bc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:34.861355 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad92867-2a5a-4281-9d4c-bc9f517cd749 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:34.908298 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 205/415] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:34.933166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb3f90c-0d48-4101-a0dc-cd5863a83fc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:34.933740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb3f90c-0d48-4101-a0dc-cd5863a83fc2 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:34.934027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb3f90c-0d48-4101-a0dc-cd5863a83fc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:34.934306 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb3f90c-0d48-4101-a0dc-cd5863a83fc2 None admin] PATH_INFO: `/v3/projects/8d6b5c1f82154555b8d48a213cdda1ed` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:34.935484 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb3f90c-0d48-4101-a0dc-cd5863a83fc2 None admin] RBAC: Authorizing `identity:get_project(project_id=8d6b5c1f82154555b8d48a213cdda1ed)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:34.937020 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb3f90c-0d48-4101-a0dc-cd5863a83fc2 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:34.939077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 211/416] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:34 2022] GET /identity/v3/projects/8d6b5c1f82154555b8d48a213cdda1ed => generated 344 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:34.945028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e376ac-0070-414b-9a9a-bbe2f0f95bc2 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:35.078146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:35.086859 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e376ac-0070-414b-9a9a-bbe2f0f95bc2 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:35.123947 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd0cb74-3a8e-406d-9eb9-1708053dede8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:35.124680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd0cb74-3a8e-406d-9eb9-1708053dede8 None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:35.124975 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd0cb74-3a8e-406d-9eb9-1708053dede8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:35.125235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd0cb74-3a8e-406d-9eb9-1708053dede8 None admin] PATH_INFO: `/v3/projects/8d6b5c1f82154555b8d48a213cdda1ed/groups/0cb97143c35a4b5baea9d1bb8ef06f53/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:35.127164 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd0cb74-3a8e-406d-9eb9-1708053dede8 None admin] RBAC: Authorizing `identity:create_grant(project_id=8d6b5c1f82154555b8d48a213cdda1ed, group_id=0cb97143c35a4b5baea9d1bb8ef06f53, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:35.128610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd0cb74-3a8e-406d-9eb9-1708053dede8 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:35.146644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 206/417] 10.222.0.26 () {58 vars in 1442 bytes} [Wed Sep 28 12:01:34 2022] PUT /identity/v3/projects/8d6b5c1f82154555b8d48a213cdda1ed/groups/0cb97143c35a4b5baea9d1bb8ef06f53/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:01:35.274191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb3f90c-0d48-4101-a0dc-cd5863a83fc2 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:35.314073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:35.418156 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb3f90c-0d48-4101-a0dc-cd5863a83fc2 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:35.470936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4124aace-f51e-4070-a454-77473924fb3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:35.471571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4124aace-f51e-4070-a454-77473924fb3a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:35.471851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4124aace-f51e-4070-a454-77473924fb3a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:35.472125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4124aace-f51e-4070-a454-77473924fb3a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:35.483667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4124aace-f51e-4070-a454-77473924fb3a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-4124aace-f51e-4070-a454-77473924fb3a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:35.492335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:35.493762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:35.494954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:35.494954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:35.494954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:35.494954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:35.494954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:35.494954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:35.494954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:35.494954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:35.494954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:35.494954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:35.494954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:35.494954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:35.494954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 212/418] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:35.500632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd0cb74-3a8e-406d-9eb9-1708053dede8 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:35.602160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd0cb74-3a8e-406d-9eb9-1708053dede8 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:35.636857 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f41360-f398-48d8-8a40-2210ef64cc39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:35.637529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f41360-f398-48d8-8a40-2210ef64cc39 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:35.637813 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f41360-f398-48d8-8a40-2210ef64cc39 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:35.638085 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f41360-f398-48d8-8a40-2210ef64cc39 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:35.638760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f41360-f398-48d8-8a40-2210ef64cc39 None admin] RBAC: Adding query filter params (name=member) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:35.639070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f41360-f398-48d8-8a40-2210ef64cc39 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:35.640474 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f41360-f398-48d8-8a40-2210ef64cc39 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:35.649772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 207/419] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:35.655543 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4124aace-f51e-4070-a454-77473924fb3a None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:35.763270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:35.782509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4124aace-f51e-4070-a454-77473924fb3a None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:35.832417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c25c08d-b5f4-4e31-a733-101d6a99c56f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:35.833599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c25c08d-b5f4-4e31-a733-101d6a99c56f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:35.833890 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c25c08d-b5f4-4e31-a733-101d6a99c56f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:35.834167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c25c08d-b5f4-4e31-a733-101d6a99c56f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:35.913303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c25c08d-b5f4-4e31-a733-101d6a99c56f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:35.914703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c25c08d-b5f4-4e31-a733-101d6a99c56f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:36.009270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:36.026116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 213/420] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:36.031671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f41360-f398-48d8-8a40-2210ef64cc39 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:36.065227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:36.157028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f41360-f398-48d8-8a40-2210ef64cc39 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:36.192892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7e23b5-6794-414f-acb1-b6d96402676f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:36.193447 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7e23b5-6794-414f-acb1-b6d96402676f None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:36.193728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7e23b5-6794-414f-acb1-b6d96402676f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:36.194005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7e23b5-6794-414f-acb1-b6d96402676f None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:36.195152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7e23b5-6794-414f-acb1-b6d96402676f None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:36.196755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7e23b5-6794-414f-acb1-b6d96402676f None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:36.198746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 208/421] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:36 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:36.204387 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c25c08d-b5f4-4e31-a733-101d6a99c56f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:36.211017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ba9f5f-4bd9-41c0-8435-2dffe3652cb9 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:36.211305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ba9f5f-4bd9-41c0-8435-2dffe3652cb9 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:36.211582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ba9f5f-4bd9-41c0-8435-2dffe3652cb9 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:36.213458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 209/422] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:01:36.218045 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef2fe9b-0c87-4846-867c-37c9f89532d1 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:36.218330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef2fe9b-0c87-4846-867c-37c9f89532d1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:36.218606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef2fe9b-0c87-4846-867c-37c9f89532d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:36.239742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-cef2fe9b-0c87-4846-867c-37c9f89532d1 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:36.241036 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-cef2fe9b-0c87-4846-867c-37c9f89532d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:36.310736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 210/423] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:36.377658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c25c08d-b5f4-4e31-a733-101d6a99c56f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:36.411096 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75985cff-2a4b-4b60-9de9-472dce7c86c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:36.411746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75985cff-2a4b-4b60-9de9-472dce7c86c2 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:36.412026 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75985cff-2a4b-4b60-9de9-472dce7c86c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:36.412301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75985cff-2a4b-4b60-9de9-472dce7c86c2 None admin] PATH_INFO: `/v3/groups/66a838cf8186454db4bcf4d63b0f0f99` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:36.413414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75985cff-2a4b-4b60-9de9-472dce7c86c2 None admin] RBAC: Authorizing `identity:get_group(group_id=66a838cf8186454db4bcf4d63b0f0f99)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:36.414851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75985cff-2a4b-4b60-9de9-472dce7c86c2 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:36.416755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 214/424] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:36 2022] GET /identity/v3/groups/66a838cf8186454db4bcf4d63b0f0f99 => generated 226 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:36.422278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef2fe9b-0c87-4846-867c-37c9f89532d1 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:36.456138 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:36.547819 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef2fe9b-0c87-4846-867c-37c9f89532d1 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:36.583420 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c223cc31-0329-492f-b1a1-ec7e2eb7f8ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:36.584104 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c223cc31-0329-492f-b1a1-ec7e2eb7f8ca None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:36.584420 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c223cc31-0329-492f-b1a1-ec7e2eb7f8ca None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:36.584727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c223cc31-0329-492f-b1a1-ec7e2eb7f8ca None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:36.585535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c223cc31-0329-492f-b1a1-ec7e2eb7f8ca None admin] RBAC: Adding query filter params (group.id=66a838cf8186454db4bcf4d63b0f0f99, role.id=615e83fab2504b6298db3efaa7c15750, scope.project.id=47fa4785636148129d33fffc2096cd4b) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:36.585879 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c223cc31-0329-492f-b1a1-ec7e2eb7f8ca None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:36.587717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c223cc31-0329-492f-b1a1-ec7e2eb7f8ca None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:36.594094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 211/425] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Sep 28 12:01:36 2022] GET /identity/v3/role_assignments?group.id=66a838cf8186454db4bcf4d63b0f0f99&scope.project.id=47fa4785636148129d33fffc2096cd4b&role.id=615e83fab2504b6298db3efaa7c15750 => generated 611 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:36.597997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75985cff-2a4b-4b60-9de9-472dce7c86c2 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:36.683193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:36.740955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75985cff-2a4b-4b60-9de9-472dce7c86c2 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:36.791296 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b67169-ded9-4586-aaf3-88f66d21b88c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:36.791943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b67169-ded9-4586-aaf3-88f66d21b88c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:36.792221 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b67169-ded9-4586-aaf3-88f66d21b88c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:36.792524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b67169-ded9-4586-aaf3-88f66d21b88c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:36.808465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b67169-ded9-4586-aaf3-88f66d21b88c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-60b67169-ded9-4586-aaf3-88f66d21b88c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:36.818845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:36.819808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:36.820783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:36.820783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:36.820783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:36.820783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:36.820783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:36.820783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:36.820783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:36.820783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:36.820783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:36.820783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:36.820783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:36.820783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:36.821891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 215/426] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:01:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:36.829639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c223cc31-0329-492f-b1a1-ec7e2eb7f8ca None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:36.833349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b430867-824d-4296-ba47-f8270d28bbbc None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:36.833652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b430867-824d-4296-ba47-f8270d28bbbc None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:36.833893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b430867-824d-4296-ba47-f8270d28bbbc None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:36.836057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 216/427] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:01:36.841976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d047739b-b46f-48a3-9300-1ece8e61065c None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:36.842281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d047739b-b46f-48a3-9300-1ece8e61065c None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:36.842514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d047739b-b46f-48a3-9300-1ece8e61065c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:36.862869 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-d047739b-b46f-48a3-9300-1ece8e61065c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:36.864129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d047739b-b46f-48a3-9300-1ece8e61065c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:36.942595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 217/428] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:36.961296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:36.984137 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c223cc31-0329-492f-b1a1-ec7e2eb7f8ca None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:37.017458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f1e6dc-a91e-4e18-b963-ae36f0c8e5ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:37.017991 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f1e6dc-a91e-4e18-b963-ae36f0c8e5ae None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:37.018266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f1e6dc-a91e-4e18-b963-ae36f0c8e5ae None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:37.018534 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f1e6dc-a91e-4e18-b963-ae36f0c8e5ae None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:37.019198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f1e6dc-a91e-4e18-b963-ae36f0c8e5ae None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:37.019500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f1e6dc-a91e-4e18-b963-ae36f0c8e5ae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:37.021404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f1e6dc-a91e-4e18-b963-ae36f0c8e5ae None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:37.029772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 212/429] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:36 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:37.035220 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d047739b-b46f-48a3-9300-1ece8e61065c None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:37.092570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:37.171811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d047739b-b46f-48a3-9300-1ece8e61065c None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:37.217061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782ac056-25c0-4d19-9a04-e889f0e2413f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:37.220077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782ac056-25c0-4d19-9a04-e889f0e2413f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:37.220394 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782ac056-25c0-4d19-9a04-e889f0e2413f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:37.220669 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782ac056-25c0-4d19-9a04-e889f0e2413f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:37.285461 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f1e6dc-a91e-4e18-b963-ae36f0c8e5ae None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:37.304012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782ac056-25c0-4d19-9a04-e889f0e2413f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:37.305701 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782ac056-25c0-4d19-9a04-e889f0e2413f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:37.359317 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:37.427200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f1e6dc-a91e-4e18-b963-ae36f0c8e5ae None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:37.432897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 218/430] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:37.434284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a138b01e-3b20-4149-81f6-5d4fa892f91a None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:37.434569 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a138b01e-3b20-4149-81f6-5d4fa892f91a None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:37.434841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a138b01e-3b20-4149-81f6-5d4fa892f91a None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:37.440331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 219/431] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:37.440331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a138b01e-3b20-4149-81f6-5d4fa892f91a None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:37.473908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389c6b10-d8aa-4677-918b-97aa353c12cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:37.474534 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389c6b10-d8aa-4677-918b-97aa353c12cf None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:37.474826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389c6b10-d8aa-4677-918b-97aa353c12cf None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:37.475088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389c6b10-d8aa-4677-918b-97aa353c12cf None admin] PATH_INFO: `/v3/roles/member` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:37.485035 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389c6b10-d8aa-4677-918b-97aa353c12cf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-389c6b10-d8aa-4677-918b-97aa353c12cf None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:37.490317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:37.491186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:37.492073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:37.492073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:37.492073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:37.492073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:37.492073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:37.492073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:37.492073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:37.492073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:37.492073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:37.492073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:37.492073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 28 12:01:37.492073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:37.495234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:37.498738 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 213/432] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:37.500279 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e06801f-6fb6-49d8-a1e8-f260055d6d73 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:37.500867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e06801f-6fb6-49d8-a1e8-f260055d6d73 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:37.501763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e06801f-6fb6-49d8-a1e8-f260055d6d73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:37.522126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-0e06801f-6fb6-49d8-a1e8-f260055d6d73 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:37.523291 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0e06801f-6fb6-49d8-a1e8-f260055d6d73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:37.565512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a138b01e-3b20-4149-81f6-5d4fa892f91a None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:37.600332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 214/433] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:37.600332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e06801f-6fb6-49d8-a1e8-f260055d6d73 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:37.622181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83ace19-3a5a-4434-ad55-63efff0510ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:37.623561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83ace19-3a5a-4434-ad55-63efff0510ee None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:37.624022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83ace19-3a5a-4434-ad55-63efff0510ee None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:37.624744 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83ace19-3a5a-4434-ad55-63efff0510ee None admin] PATH_INFO: `/v3/projects/47fa4785636148129d33fffc2096cd4b` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:37.626127 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83ace19-3a5a-4434-ad55-63efff0510ee None admin] RBAC: Authorizing `identity:get_project(project_id=47fa4785636148129d33fffc2096cd4b)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:37.629034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83ace19-3a5a-4434-ad55-63efff0510ee None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:37.631199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 220/434] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:37 2022] GET /identity/v3/projects/47fa4785636148129d33fffc2096cd4b => generated 298 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:37.639291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83ace19-3a5a-4434-ad55-63efff0510ee None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:37.668712 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:37.718789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e06801f-6fb6-49d8-a1e8-f260055d6d73 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:37.757191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:37.765838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83ace19-3a5a-4434-ad55-63efff0510ee None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:37.766511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d236f3be-6bc6-4dea-b641-fc03c23689d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:37.767318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d236f3be-6bc6-4dea-b641-fc03c23689d2 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:37.767937 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d236f3be-6bc6-4dea-b641-fc03c23689d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:37.769233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d236f3be-6bc6-4dea-b641-fc03c23689d2 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:37.770088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d236f3be-6bc6-4dea-b641-fc03c23689d2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:37.770573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d236f3be-6bc6-4dea-b641-fc03c23689d2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:37.772085 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d236f3be-6bc6-4dea-b641-fc03c23689d2 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:37.783406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 215/435] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:01:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:37.789632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d236f3be-6bc6-4dea-b641-fc03c23689d2 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:37.811294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65de11eb-fb10-4b0e-b693-8f8ca7912f2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:37.812272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de11eb-fb10-4b0e-b693-8f8ca7912f2d None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:37.812715 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de11eb-fb10-4b0e-b693-8f8ca7912f2d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:37.813115 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de11eb-fb10-4b0e-b693-8f8ca7912f2d None admin] PATH_INFO: `/v3/groups/66a838cf8186454db4bcf4d63b0f0f99` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:37.814251 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65de11eb-fb10-4b0e-b693-8f8ca7912f2d None admin] RBAC: Authorizing `identity:get_group(group_id=66a838cf8186454db4bcf4d63b0f0f99)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:37.816385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65de11eb-fb10-4b0e-b693-8f8ca7912f2d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:37.823415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 221/436] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:37 2022] GET /identity/v3/groups/66a838cf8186454db4bcf4d63b0f0f99 => generated 226 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:37.824646 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65de11eb-fb10-4b0e-b693-8f8ca7912f2d None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:37.911893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:37.915436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d236f3be-6bc6-4dea-b641-fc03c23689d2 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:37.933113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65de11eb-fb10-4b0e-b693-8f8ca7912f2d None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:37.952675 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd3920e-eec6-4f64-a4c1-a4be5cdfbc0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:37.953279 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd3920e-eec6-4f64-a4c1-a4be5cdfbc0b None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:37.953554 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd3920e-eec6-4f64-a4c1-a4be5cdfbc0b None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:37.953822 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd3920e-eec6-4f64-a4c1-a4be5cdfbc0b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:37.970162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7774d2-a5b4-4cac-8047-245c3c79347b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:37.970941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7774d2-a5b4-4cac-8047-245c3c79347b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:37.971380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7774d2-a5b4-4cac-8047-245c3c79347b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:37.971818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7774d2-a5b4-4cac-8047-245c3c79347b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:37.973139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7774d2-a5b4-4cac-8047-245c3c79347b None admin] RBAC: Adding query filter params (group.id=66a838cf8186454db4bcf4d63b0f0f99, role.id=a3902a185ac144fb8e24f4e9feb039ed, scope.project.id=47fa4785636148129d33fffc2096cd4b) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:37.973656 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7774d2-a5b4-4cac-8047-245c3c79347b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:37.975597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7774d2-a5b4-4cac-8047-245c3c79347b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:37.982720 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 222/437] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Sep 28 12:01:37 2022] GET /identity/v3/role_assignments?group.id=66a838cf8186454db4bcf4d63b0f0f99&scope.project.id=47fa4785636148129d33fffc2096cd4b&role.id=a3902a185ac144fb8e24f4e9feb039ed => generated 611 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:37.983600 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7774d2-a5b4-4cac-8047-245c3c79347b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:38.045871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:38.059955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd3920e-eec6-4f64-a4c1-a4be5cdfbc0b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:38.061573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd3920e-eec6-4f64-a4c1-a4be5cdfbc0b None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:38.133951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7774d2-a5b4-4cac-8047-245c3c79347b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:38.178158 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114f2c76-2bcf-4985-bc89-e18a386bad3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:38.178807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114f2c76-2bcf-4985-bc89-e18a386bad3e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:38.179083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114f2c76-2bcf-4985-bc89-e18a386bad3e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:38.179354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114f2c76-2bcf-4985-bc89-e18a386bad3e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:38.193298 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114f2c76-2bcf-4985-bc89-e18a386bad3e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:38.198095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 216/438] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-114f2c76-2bcf-4985-bc89-e18a386bad3e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:38.199445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:38.200581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:38.201531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:38.201531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:38.201531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:38.201531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:38.201531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:38.201531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:38.201531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:38.201531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:38.201531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:38.201531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:38.201531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:38.201531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:38.202273 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c5bdcb-935b-4a4d-a610-7ce3687149f1 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:38.202273 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c5bdcb-935b-4a4d-a610-7ce3687149f1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:38.202273 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c5bdcb-935b-4a4d-a610-7ce3687149f1 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:38.203663 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 223/439] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:01:37 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:38.205480 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 217/440] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:38 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:38.206430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114f2c76-2bcf-4985-bc89-e18a386bad3e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:38.210259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c5bdcb-935b-4a4d-a610-7ce3687149f1 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:38.211435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:38.318706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:38.328770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114f2c76-2bcf-4985-bc89-e18a386bad3e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:38.335521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c5bdcb-935b-4a4d-a610-7ce3687149f1 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:38.370983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6f1d01-a8c6-4f1e-af5e-94f5e82a6e84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:38.371617 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6f1d01-a8c6-4f1e-af5e-94f5e82a6e84 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:38.371900 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6f1d01-a8c6-4f1e-af5e-94f5e82a6e84 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:38.372447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6f1d01-a8c6-4f1e-af5e-94f5e82a6e84 None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:38.373767 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6f1d01-a8c6-4f1e-af5e-94f5e82a6e84 None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6f1d01-a8c6-4f1e-af5e-94f5e82a6e84 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:38.376300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d11bb45-9d53-405e-988e-a10faf6a0c7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:38.377035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 224/441] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:38 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:38.377141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d11bb45-9d53-405e-988e-a10faf6a0c7a None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:38.377213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d11bb45-9d53-405e-988e-a10faf6a0c7a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:38.377482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d11bb45-9d53-405e-988e-a10faf6a0c7a None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:38.377996 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a24d5b-d38b-4faf-b8f0-87f4bc80f097 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:38.378151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d11bb45-9d53-405e-988e-a10faf6a0c7a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:38.378270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a24d5b-d38b-4faf-b8f0-87f4bc80f097 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:38.378460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d11bb45-9d53-405e-988e-a10faf6a0c7a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:38.378558 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a24d5b-d38b-4faf-b8f0-87f4bc80f097 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:38.379912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d11bb45-9d53-405e-988e-a10faf6a0c7a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:38.392766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 218/442] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:38 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:38.393514 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d11bb45-9d53-405e-988e-a10faf6a0c7a None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:38.397422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-57a24d5b-d38b-4faf-b8f0-87f4bc80f097 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:38.398862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-57a24d5b-d38b-4faf-b8f0-87f4bc80f097 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:38.464022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 225/443] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:38.464716 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a24d5b-d38b-4faf-b8f0-87f4bc80f097 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:38.466847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:38.505915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d11bb45-9d53-405e-988e-a10faf6a0c7a None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:38.540583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77159c79-c502-499f-815a-616686bdfbfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:38.541212 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77159c79-c502-499f-815a-616686bdfbfd None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:38.541493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77159c79-c502-499f-815a-616686bdfbfd None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:38.541770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77159c79-c502-499f-815a-616686bdfbfd None admin] PATH_INFO: `/v3/groups/66a838cf8186454db4bcf4d63b0f0f99` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:38.542909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77159c79-c502-499f-815a-616686bdfbfd None admin] RBAC: Authorizing `identity:get_group(group_id=66a838cf8186454db4bcf4d63b0f0f99)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:38.544380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77159c79-c502-499f-815a-616686bdfbfd None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:38.546118 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 219/444] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:38 2022] GET /identity/v3/groups/66a838cf8186454db4bcf4d63b0f0f99 => generated 226 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:38.552443 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77159c79-c502-499f-815a-616686bdfbfd None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:38.571154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a24d5b-d38b-4faf-b8f0-87f4bc80f097 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:38.575878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:38.608048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9b38d8-9e6f-44e9-9f49-b17d8f362e66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:38.608684 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b38d8-9e6f-44e9-9f49-b17d8f362e66 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:38.609023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b38d8-9e6f-44e9-9f49-b17d8f362e66 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:38.609565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b38d8-9e6f-44e9-9f49-b17d8f362e66 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:38.655403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77159c79-c502-499f-815a-616686bdfbfd None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:38.678320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9b38d8-9e6f-44e9-9f49-b17d8f362e66 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:38.679789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9b38d8-9e6f-44e9-9f49-b17d8f362e66 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:38.681787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:38.690938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d35379-1e9c-4196-9166-2f8d708b90e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:38.691547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d35379-1e9c-4196-9166-2f8d708b90e9 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:38.691828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d35379-1e9c-4196-9166-2f8d708b90e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:38.692206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d35379-1e9c-4196-9166-2f8d708b90e9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:38.692990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d35379-1e9c-4196-9166-2f8d708b90e9 None admin] RBAC: Adding query filter params (group.id=66a838cf8186454db4bcf4d63b0f0f99, role.id=615e83fab2504b6298db3efaa7c15750, scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:38.693306 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d35379-1e9c-4196-9166-2f8d708b90e9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:38.694721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d35379-1e9c-4196-9166-2f8d708b90e9 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:38.700571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 220/445] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Sep 28 12:01:38 2022] GET /identity/v3/role_assignments?group.id=66a838cf8186454db4bcf4d63b0f0f99&scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4&role.id=615e83fab2504b6298db3efaa7c15750 => generated 611 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:38.737831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d35379-1e9c-4196-9166-2f8d708b90e9 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:38.797341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 226/446] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:38.802855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9b38d8-9e6f-44e9-9f49-b17d8f362e66 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:38.836912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:38.855965 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d35379-1e9c-4196-9166-2f8d708b90e9 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:38.896005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c0ab3a-d856-469e-b345-a03fd067d6da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:38.897406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c0ab3a-d856-469e-b345-a03fd067d6da None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:38.897406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c0ab3a-d856-469e-b345-a03fd067d6da None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:38.897588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c0ab3a-d856-469e-b345-a03fd067d6da None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:38.912871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c0ab3a-d856-469e-b345-a03fd067d6da None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:38.913502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9b38d8-9e6f-44e9-9f49-b17d8f362e66 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:38.915023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-80c0ab3a-d856-469e-b345-a03fd067d6da None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:38.918197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:38.920123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:38.921801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:38.921801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:38.921801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:38.921801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:38.921801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:38.921801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:38.921801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:38.921801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:38.921801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:38.921801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:38.921801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:01:38.921801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:38.921801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 221/447] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:01:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:38.925087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c0ab3a-d856-469e-b345-a03fd067d6da None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:38.956055 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7d3433-2e4f-4b5b-aeed-092a598fffbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:38.956719 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7d3433-2e4f-4b5b-aeed-092a598fffbb None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:38.956999 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7d3433-2e4f-4b5b-aeed-092a598fffbb None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:38.957268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7d3433-2e4f-4b5b-aeed-092a598fffbb None admin] PATH_INFO: `/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:38.958976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7d3433-2e4f-4b5b-aeed-092a598fffbb None admin] RBAC: Authorizing `identity:get_project(project_id=6ae10a3aea5e4ddd8b616b1e041a44d4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:38.961015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7d3433-2e4f-4b5b-aeed-092a598fffbb None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:38.962322 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 227/448] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:38 2022] GET /identity/v3/projects/6ae10a3aea5e4ddd8b616b1e041a44d4 => generated 302 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:38.967780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7d3433-2e4f-4b5b-aeed-092a598fffbb None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:39.040455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c0ab3a-d856-469e-b345-a03fd067d6da None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:39.081371 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8caca9-59a9-4610-b2d5-c2f7d3c9c1d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:39.084362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8caca9-59a9-4610-b2d5-c2f7d3c9c1d5 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:39.084362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8caca9-59a9-4610-b2d5-c2f7d3c9c1d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:39.084362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8caca9-59a9-4610-b2d5-c2f7d3c9c1d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:39.084362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8caca9-59a9-4610-b2d5-c2f7d3c9c1d5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:39.084362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8caca9-59a9-4610-b2d5-c2f7d3c9c1d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:39.084724 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7d3433-2e4f-4b5b-aeed-092a598fffbb None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:39.085418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8caca9-59a9-4610-b2d5-c2f7d3c9c1d5 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:39.094966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 222/449] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:39.100688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8caca9-59a9-4610-b2d5-c2f7d3c9c1d5 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:39.111135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:39.118802 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844ad33c-8019-48db-b839-b19ff4800fb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:39.119550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844ad33c-8019-48db-b839-b19ff4800fb7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:39.119829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844ad33c-8019-48db-b839-b19ff4800fb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:39.120112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844ad33c-8019-48db-b839-b19ff4800fb7 None admin] PATH_INFO: `/v3/groups/66a838cf8186454db4bcf4d63b0f0f99` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:39.121454 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844ad33c-8019-48db-b839-b19ff4800fb7 None admin] RBAC: Authorizing `identity:get_group(group_id=66a838cf8186454db4bcf4d63b0f0f99)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:39.122876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844ad33c-8019-48db-b839-b19ff4800fb7 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:39.124431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 228/450] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:38 2022] GET /identity/v3/groups/66a838cf8186454db4bcf4d63b0f0f99 => generated 226 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:39.130818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844ad33c-8019-48db-b839-b19ff4800fb7 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:39.168054 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:39.199374 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8caca9-59a9-4610-b2d5-c2f7d3c9c1d5 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:39.228161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844ad33c-8019-48db-b839-b19ff4800fb7 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:39.233296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368b1e08-845d-4990-8069-10c0dc81b1e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:39.233906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b1e08-845d-4990-8069-10c0dc81b1e0 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:39.234191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b1e08-845d-4990-8069-10c0dc81b1e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:39.234470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b1e08-845d-4990-8069-10c0dc81b1e0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:39.261411 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a15fb6-7cd6-47a4-a942-bb15fd6ca986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:39.262006 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a15fb6-7cd6-47a4-a942-bb15fd6ca986 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:39.262287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a15fb6-7cd6-47a4-a942-bb15fd6ca986 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:39.262560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a15fb6-7cd6-47a4-a942-bb15fd6ca986 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:39.263282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a15fb6-7cd6-47a4-a942-bb15fd6ca986 None admin] RBAC: Adding query filter params (group.id=66a838cf8186454db4bcf4d63b0f0f99, role.id=a3902a185ac144fb8e24f4e9feb039ed, scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:39.263593 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a15fb6-7cd6-47a4-a942-bb15fd6ca986 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:39.265058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a15fb6-7cd6-47a4-a942-bb15fd6ca986 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:39.271097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 229/451] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Sep 28 12:01:39 2022] GET /identity/v3/role_assignments?group.id=66a838cf8186454db4bcf4d63b0f0f99&scope.project.id=6ae10a3aea5e4ddd8b616b1e041a44d4&role.id=a3902a185ac144fb8e24f4e9feb039ed => generated 611 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:39.302621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368b1e08-845d-4990-8069-10c0dc81b1e0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:39.304113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368b1e08-845d-4990-8069-10c0dc81b1e0 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:39.315855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:39.414156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 223/452] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 28 12:01:39.420077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a15fb6-7cd6-47a4-a942-bb15fd6ca986 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:39.522522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a15fb6-7cd6-47a4-a942-bb15fd6ca986 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:39.524043 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:39.557313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d2aadc-c1b9-4822-8c82-c6d39ef8c521 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:39.557942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d2aadc-c1b9-4822-8c82-c6d39ef8c521 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:39.558242 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d2aadc-c1b9-4822-8c82-c6d39ef8c521 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:39.558489 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d2aadc-c1b9-4822-8c82-c6d39ef8c521 None admin] PATH_INFO: `/v3/projects/8d6b5c1f82154555b8d48a213cdda1ed` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:39.559612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d2aadc-c1b9-4822-8c82-c6d39ef8c521 None admin] RBAC: Authorizing `identity:get_project(project_id=8d6b5c1f82154555b8d48a213cdda1ed)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:39.561086 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d2aadc-c1b9-4822-8c82-c6d39ef8c521 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:39.562696 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 230/453] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:01:39 2022] GET /identity/v3/projects/8d6b5c1f82154555b8d48a213cdda1ed => generated 344 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:39.569131 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368b1e08-845d-4990-8069-10c0dc81b1e0 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:39.667192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368b1e08-845d-4990-8069-10c0dc81b1e0 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:39.676053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:39.700238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988dc9b6-99f5-4e2d-80b1-8c834c0d5b1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:39.700890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988dc9b6-99f5-4e2d-80b1-8c834c0d5b1a None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:39.701172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988dc9b6-99f5-4e2d-80b1-8c834c0d5b1a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:39.701448 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988dc9b6-99f5-4e2d-80b1-8c834c0d5b1a None admin] PATH_INFO: `/v3/groups/0cb97143c35a4b5baea9d1bb8ef06f53` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:39.702551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988dc9b6-99f5-4e2d-80b1-8c834c0d5b1a None admin] RBAC: Authorizing `identity:get_group(group_id=0cb97143c35a4b5baea9d1bb8ef06f53)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:39.703971 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988dc9b6-99f5-4e2d-80b1-8c834c0d5b1a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:39.705549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 224/454] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Sep 28 12:01:39 2022] GET /identity/v3/groups/0cb97143c35a4b5baea9d1bb8ef06f53 => generated 229 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:39.712351 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d2aadc-c1b9-4822-8c82-c6d39ef8c521 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:39.807263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d2aadc-c1b9-4822-8c82-c6d39ef8c521 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:39.840065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053591f6-4cbb-4efc-ad46-b0cce2004248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:39.840681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053591f6-4cbb-4efc-ad46-b0cce2004248 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:39.840969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053591f6-4cbb-4efc-ad46-b0cce2004248 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:39.841241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053591f6-4cbb-4efc-ad46-b0cce2004248 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:39.841960 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053591f6-4cbb-4efc-ad46-b0cce2004248 None admin] RBAC: Adding query filter params (group.id=0cb97143c35a4b5baea9d1bb8ef06f53, role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=8d6b5c1f82154555b8d48a213cdda1ed) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:39.842269 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053591f6-4cbb-4efc-ad46-b0cce2004248 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:39.843707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053591f6-4cbb-4efc-ad46-b0cce2004248 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:39.849323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 231/455] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Sep 28 12:01:39 2022] GET /identity/v3/role_assignments?group.id=0cb97143c35a4b5baea9d1bb8ef06f53&scope.project.id=8d6b5c1f82154555b8d48a213cdda1ed&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 611 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:42.627517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56b317c-167d-4dd0-ba40-adb69440f3e1 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:42.628380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56b317c-167d-4dd0-ba40-adb69440f3e1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:42.628794 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56b317c-167d-4dd0-ba40-adb69440f3e1 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:42.631096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 225/456] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:42 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:42.636235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87829ccb-564b-43af-80b9-bb438cd1516d None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:42.636652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87829ccb-564b-43af-80b9-bb438cd1516d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:42.636913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87829ccb-564b-43af-80b9-bb438cd1516d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:42.656379 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-87829ccb-564b-43af-80b9-bb438cd1516d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:42.657567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-87829ccb-564b-43af-80b9-bb438cd1516d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:42.684660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:42.728675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 232/457] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:42.766366 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8af650-3786-4071-b1b0-14c553375bff None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:42.766643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8af650-3786-4071-b1b0-14c553375bff None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:42.766837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8af650-3786-4071-b1b0-14c553375bff None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:42.768430 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 226/458] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:42.773725 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cd3e22-1536-4542-a12b-8d52cf6e594e None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:42.774006 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cd3e22-1536-4542-a12b-8d52cf6e594e None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:42.774277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cd3e22-1536-4542-a12b-8d52cf6e594e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:42.797338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-f4cd3e22-1536-4542-a12b-8d52cf6e594e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:42.798838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f4cd3e22-1536-4542-a12b-8d52cf6e594e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:42.897770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 233/459] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:42.914099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cce3db1-ecb7-4717-8c66-cc7b11039f8d None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:42.914377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cce3db1-ecb7-4717-8c66-cc7b11039f8d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:42.914873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cce3db1-ecb7-4717-8c66-cc7b11039f8d None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:42.916613 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 227/460] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:42.921633 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4c4acb-1774-4337-91d4-78eb0c6a0e4d None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:42.921914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4c4acb-1774-4337-91d4-78eb0c6a0e4d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:42.922180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4c4acb-1774-4337-91d4-78eb0c6a0e4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:42.946075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-0b4c4acb-1774-4337-91d4-78eb0c6a0e4d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:42.947244 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0b4c4acb-1774-4337-91d4-78eb0c6a0e4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:43.044895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 234/461] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:43.070225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cce3db1-ecb7-4717-8c66-cc7b11039f8d None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:43.178858 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cce3db1-ecb7-4717-8c66-cc7b11039f8d None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:43.197113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6a91b0-1424-4a8f-91a2-2b0f4a261c57 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:43.197402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6a91b0-1424-4a8f-91a2-2b0f4a261c57 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:43.197681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6a91b0-1424-4a8f-91a2-2b0f4a261c57 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:43.199262 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 235/462] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:43.204246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa88d065-dc38-4a3d-8020-5f18e6f0e43b None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:43.204563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa88d065-dc38-4a3d-8020-5f18e6f0e43b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:43.204836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa88d065-dc38-4a3d-8020-5f18e6f0e43b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:43.218365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9a6263-7761-4cdf-adc6-247a98c06ede None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:43.219838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9a6263-7761-4cdf-adc6-247a98c06ede None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:43.220665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9a6263-7761-4cdf-adc6-247a98c06ede None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:43.222626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9a6263-7761-4cdf-adc6-247a98c06ede None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:43.243739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9a6263-7761-4cdf-adc6-247a98c06ede None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:43.246154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-fa88d065-dc38-4a3d-8020-5f18e6f0e43b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:43.247932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9a6263-7761-4cdf-adc6-247a98c06ede None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:43.248790 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-fa88d065-dc38-4a3d-8020-5f18e6f0e43b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:43.269551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-8f9a6263-7761-4cdf-adc6-247a98c06ede None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:43.274024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:43.274940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:43.275758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:43.275758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:43.275758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:43.275758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 228/463] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:43.294300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9a6263-7761-4cdf-adc6-247a98c06ede None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:43.337330 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 236/464] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:43.344496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:43.377683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa88d065-dc38-4a3d-8020-5f18e6f0e43b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:43.429435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9a6263-7761-4cdf-adc6-247a98c06ede None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:43.467934 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4dd5b4-c302-4771-b645-7b4e4e4bdf62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:43.468756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4dd5b4-c302-4771-b645-7b4e4e4bdf62 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:43.469128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4dd5b4-c302-4771-b645-7b4e4e4bdf62 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:43.469500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4dd5b4-c302-4771-b645-7b4e4e4bdf62 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:43.470405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4dd5b4-c302-4771-b645-7b4e4e4bdf62 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:43.470809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4dd5b4-c302-4771-b645-7b4e4e4bdf62 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:43.472276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4dd5b4-c302-4771-b645-7b4e4e4bdf62 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:43.485755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa88d065-dc38-4a3d-8020-5f18e6f0e43b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:43.487062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 229/465] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:43.493319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4dd5b4-c302-4771-b645-7b4e4e4bdf62 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:43.534502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153668a2-544d-4119-ae18-a8ce15438432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:43.535382 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153668a2-544d-4119-ae18-a8ce15438432 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:43.536346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153668a2-544d-4119-ae18-a8ce15438432 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:43.536759 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153668a2-544d-4119-ae18-a8ce15438432 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:43.550988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153668a2-544d-4119-ae18-a8ce15438432 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:43.553816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153668a2-544d-4119-ae18-a8ce15438432 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-153668a2-544d-4119-ae18-a8ce15438432 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:43.574846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:43.575780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:43.576647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:43.576647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:43.576647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:43.584131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 237/466] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:43.585158 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153668a2-544d-4119-ae18-a8ce15438432 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:43.595468 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:43.599582 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:43.617997 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4dd5b4-c302-4771-b645-7b4e4e4bdf62 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:43.658418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43be8f5b-48ee-4db4-9aa3-a07c46caeacc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:43.659045 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43be8f5b-48ee-4db4-9aa3-a07c46caeacc None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:43.659313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43be8f5b-48ee-4db4-9aa3-a07c46caeacc None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:43.659586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43be8f5b-48ee-4db4-9aa3-a07c46caeacc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:43.671226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43be8f5b-48ee-4db4-9aa3-a07c46caeacc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:43.672692 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43be8f5b-48ee-4db4-9aa3-a07c46caeacc None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-43be8f5b-48ee-4db4-9aa3-a07c46caeacc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:43.686774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:43.687687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:43.688915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:43.688915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:43.688915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:43.688915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 230/467] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:43.688915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43be8f5b-48ee-4db4-9aa3-a07c46caeacc None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:43.707424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153668a2-544d-4119-ae18-a8ce15438432 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:43.743897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe92414-a47d-4810-99bc-b67e0c3115e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:43.744522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe92414-a47d-4810-99bc-b67e0c3115e0 None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:43.744796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe92414-a47d-4810-99bc-b67e0c3115e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:43.745069 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe92414-a47d-4810-99bc-b67e0c3115e0 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:43.745711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe92414-a47d-4810-99bc-b67e0c3115e0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:43.747154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe92414-a47d-4810-99bc-b67e0c3115e0 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:43.768445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 238/468] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:01:43 2022] POST /identity/v3/users => generated 246 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:01:43.769079 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe92414-a47d-4810-99bc-b67e0c3115e0 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:43.814751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43be8f5b-48ee-4db4-9aa3-a07c46caeacc None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:43.850794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:43.860696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df1bcd8-657a-42be-989d-4236d15c9696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:43.861046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df1bcd8-657a-42be-989d-4236d15c9696 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:43.861238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df1bcd8-657a-42be-989d-4236d15c9696 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:43.861486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df1bcd8-657a-42be-989d-4236d15c9696 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:43.862153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df1bcd8-657a-42be-989d-4236d15c9696 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:43.862485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df1bcd8-657a-42be-989d-4236d15c9696 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:43.864530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df1bcd8-657a-42be-989d-4236d15c9696 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:43.874928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:43.879003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 231/469] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:43.880661 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df1bcd8-657a-42be-989d-4236d15c9696 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:43.906408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe92414-a47d-4810-99bc-b67e0c3115e0 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:43.946601 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b238f6cc-ad76-48ba-9ad1-163af7f2876e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:43.947249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b238f6cc-ad76-48ba-9ad1-163af7f2876e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:43.947528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b238f6cc-ad76-48ba-9ad1-163af7f2876e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:43.947800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b238f6cc-ad76-48ba-9ad1-163af7f2876e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:43.960351 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b238f6cc-ad76-48ba-9ad1-163af7f2876e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:43.961834 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b238f6cc-ad76-48ba-9ad1-163af7f2876e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-b238f6cc-ad76-48ba-9ad1-163af7f2876e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:43.978295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:43.979208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:43.980722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:43.980722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:43.980722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:43.980722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 239/470] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:43.980722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b238f6cc-ad76-48ba-9ad1-163af7f2876e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:43.997646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df1bcd8-657a-42be-989d-4236d15c9696 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:44.041594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b82699-f9f5-477d-835e-72ae2be0d54e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:44.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b82699-f9f5-477d-835e-72ae2be0d54e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:44.042470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b82699-f9f5-477d-835e-72ae2be0d54e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:44.042798 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b82699-f9f5-477d-835e-72ae2be0d54e None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:44.043455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b82699-f9f5-477d-835e-72ae2be0d54e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:44.043756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b82699-f9f5-477d-835e-72ae2be0d54e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:44.045142 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b82699-f9f5-477d-835e-72ae2be0d54e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:44.057098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 232/471] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:44.062147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb616ef-d01a-45d2-abe1-bcb302707c54 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:44.062147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb616ef-d01a-45d2-abe1-bcb302707c54 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:44.062147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb616ef-d01a-45d2-abe1-bcb302707c54 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:44.062147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 233/472] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:44.062147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb616ef-d01a-45d2-abe1-bcb302707c54 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:44.099675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b238f6cc-ad76-48ba-9ad1-163af7f2876e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:44.114504 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:44.126839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:44.136623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ee0f25-908a-43d1-bd40-1c905fa0aafd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:44.137339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ee0f25-908a-43d1-bd40-1c905fa0aafd None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:44.137629 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ee0f25-908a-43d1-bd40-1c905fa0aafd None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:44.137899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ee0f25-908a-43d1-bd40-1c905fa0aafd None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:44.138548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ee0f25-908a-43d1-bd40-1c905fa0aafd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:44.140342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ee0f25-908a-43d1-bd40-1c905fa0aafd None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:44.162655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 240/473] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:01:43 2022] POST /identity/v3/users => generated 248 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:01:44.163460 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ee0f25-908a-43d1-bd40-1c905fa0aafd None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:44.175682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb616ef-d01a-45d2-abe1-bcb302707c54 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:44.218131 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0c1fcc-2a86-4061-9fbe-b0eb959a0e17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:44.218907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0c1fcc-2a86-4061-9fbe-b0eb959a0e17 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:44.219189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0c1fcc-2a86-4061-9fbe-b0eb959a0e17 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:44.219515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0c1fcc-2a86-4061-9fbe-b0eb959a0e17 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:44.220391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0c1fcc-2a86-4061-9fbe-b0eb959a0e17 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:44.220764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0c1fcc-2a86-4061-9fbe-b0eb959a0e17 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:44.222303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0c1fcc-2a86-4061-9fbe-b0eb959a0e17 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:44.241569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 234/474] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:44.241569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cd5f54-0c16-477f-b2fe-355528c5c329 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:44.241569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cd5f54-0c16-477f-b2fe-355528c5c329 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:44.241569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cd5f54-0c16-477f-b2fe-355528c5c329 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:44.260707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-e2cd5f54-0c16-477f-b2fe-355528c5c329 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:44.262117 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e2cd5f54-0c16-477f-b2fe-355528c5c329 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:44.329930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ee0f25-908a-43d1-bd40-1c905fa0aafd None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:44.350287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 235/475] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:44.351465 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cd5f54-0c16-477f-b2fe-355528c5c329 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:44.371912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d5f54e-48d7-4bf9-a860-405518c250f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:44.372464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d5f54e-48d7-4bf9-a860-405518c250f0 None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:44.372743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d5f54e-48d7-4bf9-a860-405518c250f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:44.373069 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d5f54e-48d7-4bf9-a860-405518c250f0 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:44.373745 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d5f54e-48d7-4bf9-a860-405518c250f0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:44.375194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d5f54e-48d7-4bf9-a860-405518c250f0 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:44.403541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 241/476] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:01:44 2022] POST /identity/v3/users => generated 248 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:01:44.433210 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:44.502270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cd5f54-0c16-477f-b2fe-355528c5c329 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:44.541418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aad31b3-e3f3-4d22-9514-ef3e06507535 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:44.542087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aad31b3-e3f3-4d22-9514-ef3e06507535 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:44.542374 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aad31b3-e3f3-4d22-9514-ef3e06507535 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:44.542697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aad31b3-e3f3-4d22-9514-ef3e06507535 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:44.543362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aad31b3-e3f3-4d22-9514-ef3e06507535 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:44.544930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aad31b3-e3f3-4d22-9514-ef3e06507535 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:44.572873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 236/477] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:01:44 2022] POST /identity/v3/users => generated 249 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:01:44.671447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d5f54e-48d7-4bf9-a860-405518c250f0 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:44.709421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:44.797636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d5f54e-48d7-4bf9-a860-405518c250f0 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:44.838271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0286a1b-f712-4d07-a6fa-18e9dbbf6d83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:44.839174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0286a1b-f712-4d07-a6fa-18e9dbbf6d83 None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:44.839653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0286a1b-f712-4d07-a6fa-18e9dbbf6d83 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:44.839925 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0286a1b-f712-4d07-a6fa-18e9dbbf6d83 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:44.840653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0286a1b-f712-4d07-a6fa-18e9dbbf6d83 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:44.842064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0286a1b-f712-4d07-a6fa-18e9dbbf6d83 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:44.851123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.sql.core [None req-a0286a1b-f712-4d07-a6fa-18e9dbbf6d83 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 28 12:01:44.851123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 28 12:01:44.851123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [parameters: {'id': '6d30e03ca4934ddc8735cd80c5b63b39', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 28 12:01:44.851123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81592) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-a0286a1b-f712-4d07-a6fa-18e9dbbf6d83 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application result = self._query(query) Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application conn.query(q) Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application result.read() Sep 28 12:01:44.868356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return ref.to_dict() Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 28 12:01:44.869301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application next(self.gen) Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application yield resource Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application next(self.gen) Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self.session.rollback() Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self.force_reraise() Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise self.value Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application session.commit() Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 28 12:01:44.870203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._prepare_impl() Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self.session.flush() Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._flush(objects) Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application compat.raise_( Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application flush_context.execute() Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rec.execute(self) Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 28 12:01:44.871044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 28 12:01:44.871958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application result = self._query(query) Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application conn.query(q) Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application result.read() Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 28 12:01:44.872814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [parameters: {'id': '6d30e03ca4934ddc8735cd80c5b63b39', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:44.873590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 28 12:01:44.874479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:44.875316 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 242/478] 10.222.0.26 () {60 vars in 1131 bytes} [Wed Sep 28 12:01:44 2022] POST /identity/v3/roles => generated 144 bytes in 199 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:44.875381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aad31b3-e3f3-4d22-9514-ef3e06507535 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:44.986415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aad31b3-e3f3-4d22-9514-ef3e06507535 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:44.994985 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:45.041041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b37b52-de24-4cf7-8a75-cf8b6c6a48f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:45.041690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b37b52-de24-4cf7-8a75-cf8b6c6a48f3 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:45.041968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b37b52-de24-4cf7-8a75-cf8b6c6a48f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:45.042238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b37b52-de24-4cf7-8a75-cf8b6c6a48f3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:45.056113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b37b52-de24-4cf7-8a75-cf8b6c6a48f3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-06b37b52-de24-4cf7-8a75-cf8b6c6a48f3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:45.060930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:45.061796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:45.062579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:45.062579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:45.062579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:45.062579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:45.062579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:45.062579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:45.062579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:45.062579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:45.062579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:45.062579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:45.062579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 28 12:01:45.062579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:45.062579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 237/479] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:01:44 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:45.068704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0286a1b-f712-4d07-a6fa-18e9dbbf6d83 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:45.179202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:45.193662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0286a1b-f712-4d07-a6fa-18e9dbbf6d83 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:45.229313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5dd5f4f-c6e5-4eaf-9993-ea02b53e1a6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:45.229917 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dd5f4f-c6e5-4eaf-9993-ea02b53e1a6f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:45.230195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dd5f4f-c6e5-4eaf-9993-ea02b53e1a6f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:45.230465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dd5f4f-c6e5-4eaf-9993-ea02b53e1a6f None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:45.231131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5dd5f4f-c6e5-4eaf-9993-ea02b53e1a6f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:45.231434 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5dd5f4f-c6e5-4eaf-9993-ea02b53e1a6f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:45.232842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5dd5f4f-c6e5-4eaf-9993-ea02b53e1a6f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:45.241555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 243/480] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:45 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:46.355332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01742b78-dcc0-4c1b-9572-7184077cb18c None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:46.355844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01742b78-dcc0-4c1b-9572-7184077cb18c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:46.356135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01742b78-dcc0-4c1b-9572-7184077cb18c None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:46.357663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 238/481] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:46.362955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c23d83-7561-4584-aa4a-568c012ef86a None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:46.363347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c23d83-7561-4584-aa4a-568c012ef86a None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:46.363533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c23d83-7561-4584-aa4a-568c012ef86a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:46.382846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-20c23d83-7561-4584-aa4a-568c012ef86a None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:46.384037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-20c23d83-7561-4584-aa4a-568c012ef86a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:46.468774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 244/482] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:46.828747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01742b78-dcc0-4c1b-9572-7184077cb18c None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:46.958442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01742b78-dcc0-4c1b-9572-7184077cb18c None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:47.010353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44cc155e-d15f-447a-bc6b-8d3934acda8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:47.011019 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cc155e-d15f-447a-bc6b-8d3934acda8b None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:47.011302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cc155e-d15f-447a-bc6b-8d3934acda8b None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:47.011574 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cc155e-d15f-447a-bc6b-8d3934acda8b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:47.027231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44cc155e-d15f-447a-bc6b-8d3934acda8b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-44cc155e-d15f-447a-bc6b-8d3934acda8b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:47.032023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:47.032924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:47.033687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:47.033687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:47.033687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:47.033687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:47.033687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:47.033687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:47.033687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:47.033687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:47.033687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:47.033687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:47.033687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:47.033687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:47.033687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 239/483] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:01:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:47.039954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c23d83-7561-4584-aa4a-568c012ef86a None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:47.126783 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d07ada3-55fa-4ce4-a231-215b902f96fa None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:47.127090 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d07ada3-55fa-4ce4-a231-215b902f96fa None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:47.127308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d07ada3-55fa-4ce4-a231-215b902f96fa None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:47.129022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 240/484] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:01:47.138538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe086b3b-fae7-4753-a465-9f0379b94ef8 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:47.138828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe086b3b-fae7-4753-a465-9f0379b94ef8 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:47.139094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe086b3b-fae7-4753-a465-9f0379b94ef8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:47.171387 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:47.172666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-fe086b3b-fae7-4753-a465-9f0379b94ef8 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:47.174016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-fe086b3b-fae7-4753-a465-9f0379b94ef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:47.193013 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c23d83-7561-4584-aa4a-568c012ef86a None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:47.221776 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:47.257723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c9a3a7-5a46-4681-9dcb-ff16bd60beb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:47.258666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c9a3a7-5a46-4681-9dcb-ff16bd60beb9 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:47.259125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c9a3a7-5a46-4681-9dcb-ff16bd60beb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:47.259575 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c9a3a7-5a46-4681-9dcb-ff16bd60beb9 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:47.260481 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c9a3a7-5a46-4681-9dcb-ff16bd60beb9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:47.260983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c9a3a7-5a46-4681-9dcb-ff16bd60beb9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:47.263282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c9a3a7-5a46-4681-9dcb-ff16bd60beb9 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:47.275609 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 245/485] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:01:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:47.277089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4c04fc-8874-4b9d-9307-c7db19d93166 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:47.277545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4c04fc-8874-4b9d-9307-c7db19d93166 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:47.277967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4c04fc-8874-4b9d-9307-c7db19d93166 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:47.279324 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 246/486] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:47.280975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4c04fc-8874-4b9d-9307-c7db19d93166 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:47.299184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 241/487] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:47.300958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb9adce-7eeb-4653-b72e-76c00c788c1e None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:47.302093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb9adce-7eeb-4653-b72e-76c00c788c1e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:47.302583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb9adce-7eeb-4653-b72e-76c00c788c1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:47.322289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-2bb9adce-7eeb-4653-b72e-76c00c788c1e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:47.323904 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2bb9adce-7eeb-4653-b72e-76c00c788c1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:47.418641 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 242/488] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:47.466653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4c04fc-8874-4b9d-9307-c7db19d93166 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:47.515474 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66f2a8c-7653-4d26-bce2-a7b217dda290 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:47.516167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66f2a8c-7653-4d26-bce2-a7b217dda290 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:47.516539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66f2a8c-7653-4d26-bce2-a7b217dda290 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:47.517025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66f2a8c-7653-4d26-bce2-a7b217dda290 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:47.532025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66f2a8c-7653-4d26-bce2-a7b217dda290 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:47.533772 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66f2a8c-7653-4d26-bce2-a7b217dda290 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:47.539316 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:47.545000 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5d79b2-9abb-485e-8041-c8974b6ce610 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:47.545341 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5d79b2-9abb-485e-8041-c8974b6ce610 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:47.545620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5d79b2-9abb-485e-8041-c8974b6ce610 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:47.547761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 243/489] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:47.553467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032ea344-3d72-4916-a84d-6d8090196d89 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:47.553807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032ea344-3d72-4916-a84d-6d8090196d89 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:47.554088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032ea344-3d72-4916-a84d-6d8090196d89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-a66f2a8c-7653-4d26-bce2-a7b217dda290 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:47.556835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:47.557704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:47.558610 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:47.558610 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:47.558610 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:47.559150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 247/490] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:47.563512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66f2a8c-7653-4d26-bce2-a7b217dda290 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:47.578787 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-032ea344-3d72-4916-a84d-6d8090196d89 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:47.579964 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-032ea344-3d72-4916-a84d-6d8090196d89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:47.677333 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 244/491] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:47.678117 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032ea344-3d72-4916-a84d-6d8090196d89 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:47.708151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:47.718852 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66f2a8c-7653-4d26-bce2-a7b217dda290 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:47.776289 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eefb721-7280-4d55-8991-a3648851072b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:47.777535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eefb721-7280-4d55-8991-a3648851072b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:47.778269 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eefb721-7280-4d55-8991-a3648851072b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:47.779155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eefb721-7280-4d55-8991-a3648851072b None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:47.780279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eefb721-7280-4d55-8991-a3648851072b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:47.780827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eefb721-7280-4d55-8991-a3648851072b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:47.782685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eefb721-7280-4d55-8991-a3648851072b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:47.799216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 248/492] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:47.801138 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eefb721-7280-4d55-8991-a3648851072b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:47.825236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032ea344-3d72-4916-a84d-6d8090196d89 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:47.866928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7029828f-41d0-4f13-8899-44141b2e6e84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:47.867580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7029828f-41d0-4f13-8899-44141b2e6e84 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:47.868218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7029828f-41d0-4f13-8899-44141b2e6e84 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:47.868539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7029828f-41d0-4f13-8899-44141b2e6e84 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:47.877099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:47.880856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7029828f-41d0-4f13-8899-44141b2e6e84 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-7029828f-41d0-4f13-8899-44141b2e6e84 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:47.886520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:47.887402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:47.888209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:47.888209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:47.888209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:47.888209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:47.888209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:47.888209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:47.888209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:47.888209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:47.888209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:47.888209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:47.888209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:47.888209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:47.888209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 245/493] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:01:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:47.888892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7029828f-41d0-4f13-8899-44141b2e6e84 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:47.945816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eefb721-7280-4d55-8991-a3648851072b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:47.977073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:47.989630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223b3b95-6fbb-479a-b8d7-12a52141eb04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:47.990268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223b3b95-6fbb-479a-b8d7-12a52141eb04 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:47.990549 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223b3b95-6fbb-479a-b8d7-12a52141eb04 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:47.990823 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223b3b95-6fbb-479a-b8d7-12a52141eb04 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:48.003256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223b3b95-6fbb-479a-b8d7-12a52141eb04 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:48.003407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7029828f-41d0-4f13-8899-44141b2e6e84 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-223b3b95-6fbb-479a-b8d7-12a52141eb04 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:48.010292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:48.011162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:48.011939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:48.011939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:48.011939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:48.011939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:48.011939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:48.011939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:48.011939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:48.011939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:48.011939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:48.011939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:48.011939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:48.011939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:48.012555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 249/494] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:01:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:48.013559 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223b3b95-6fbb-479a-b8d7-12a52141eb04 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:48.044140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341f5614-38d0-492f-ab06-6a7cdbef67eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:48.044783 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341f5614-38d0-492f-ab06-6a7cdbef67eb None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:48.045063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341f5614-38d0-492f-ab06-6a7cdbef67eb None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:48.045339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341f5614-38d0-492f-ab06-6a7cdbef67eb None admin] PATH_INFO: `/v3/users/nova` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:48.053050 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341f5614-38d0-492f-ab06-6a7cdbef67eb None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:48.054493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341f5614-38d0-492f-ab06-6a7cdbef67eb None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-341f5614-38d0-492f-ab06-6a7cdbef67eb None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:01:48.068140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:48.069232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:48.070408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:48.070408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:48.070408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:01:48.070408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:01:48.070408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:01:48.070408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:01:48.070408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:01:48.070408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:01:48.070408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:01:48.070408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:48.073855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 246/495] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Sep 28 12:01:47 2022] GET /identity/v3/users/nova => generated 82 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:48.074553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341f5614-38d0-492f-ab06-6a7cdbef67eb None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:48.123625 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223b3b95-6fbb-479a-b8d7-12a52141eb04 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:48.140606 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:48.159206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eeee175-9a2a-4884-877a-7543e717b98f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:48.159801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eeee175-9a2a-4884-877a-7543e717b98f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:48.160082 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eeee175-9a2a-4884-877a-7543e717b98f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:48.160378 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eeee175-9a2a-4884-877a-7543e717b98f None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:48.161077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eeee175-9a2a-4884-877a-7543e717b98f None admin] RBAC: Adding query filter params (name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:48.161389 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eeee175-9a2a-4884-877a-7543e717b98f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:48.162776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eeee175-9a2a-4884-877a-7543e717b98f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:48.170833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 250/496] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:01:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:48.171641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eeee175-9a2a-4884-877a-7543e717b98f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:48.172660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341f5614-38d0-492f-ab06-6a7cdbef67eb None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:48.206953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f89e4f-a4f4-4579-8701-80945c81922b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:48.207507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f89e4f-a4f4-4579-8701-80945c81922b None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:48.207779 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f89e4f-a4f4-4579-8701-80945c81922b None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:48.208050 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f89e4f-a4f4-4579-8701-80945c81922b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:48.210632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:48.219677 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f89e4f-a4f4-4579-8701-80945c81922b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-f4f89e4f-a4f4-4579-8701-80945c81922b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:48.225205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:48.226054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:48.226831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:48.226831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:48.226831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:48.226831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:48.226831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:48.226831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:48.226831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:48.226831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:48.226831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:48.226831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:48.226831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:48.226831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:48.226831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 247/497] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:01:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:48.227458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f89e4f-a4f4-4579-8701-80945c81922b None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:48.272320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eeee175-9a2a-4884-877a-7543e717b98f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:48.305747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd90d83-3a56-4f60-b0e1-75bd65f39fc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:48.306342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd90d83-3a56-4f60-b0e1-75bd65f39fc4 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:48.306616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd90d83-3a56-4f60-b0e1-75bd65f39fc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:48.306883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd90d83-3a56-4f60-b0e1-75bd65f39fc4 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:48.307535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd90d83-3a56-4f60-b0e1-75bd65f39fc4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:48.307833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd90d83-3a56-4f60-b0e1-75bd65f39fc4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:48.309222 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd90d83-3a56-4f60-b0e1-75bd65f39fc4 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:48.316775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 251/498] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:01:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:48.317995 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c4ae2a-8601-4f1d-94ec-5fecab777f11 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:48.318280 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c4ae2a-8601-4f1d-94ec-5fecab777f11 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:48.318554 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c4ae2a-8601-4f1d-94ec-5fecab777f11 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:48.319734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 252/499] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:48 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:48.320445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c4ae2a-8601-4f1d-94ec-5fecab777f11 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:48.327564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f89e4f-a4f4-4579-8701-80945c81922b None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:48.361364 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adb659d-c626-41ba-b200-648a522dc0ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:48.361967 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adb659d-c626-41ba-b200-648a522dc0ba None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:48.362244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adb659d-c626-41ba-b200-648a522dc0ba None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:48.362516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adb659d-c626-41ba-b200-648a522dc0ba None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:48.362831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:48.363252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adb659d-c626-41ba-b200-648a522dc0ba None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:48.363558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adb659d-c626-41ba-b200-648a522dc0ba None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:48.365755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adb659d-c626-41ba-b200-648a522dc0ba None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:48.401444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 248/500] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Sep 28 12:01:48 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:48.402257 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adb659d-c626-41ba-b200-648a522dc0ba None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:48.422017 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c4ae2a-8601-4f1d-94ec-5fecab777f11 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:48.455403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a5f4e0-225f-4290-b12d-a945b6a8af57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:48.456022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a5f4e0-225f-4290-b12d-a945b6a8af57 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:48.456314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a5f4e0-225f-4290-b12d-a945b6a8af57 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:48.456571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:48.457058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a5f4e0-225f-4290-b12d-a945b6a8af57 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:48.467200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a5f4e0-225f-4290-b12d-a945b6a8af57 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:48.468694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a5f4e0-225f-4290-b12d-a945b6a8af57 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-92a5f4e0-225f-4290-b12d-a945b6a8af57 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:48.482415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:48.483309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:48.484130 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:48.484130 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:48.484130 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:48.484130 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 253/501] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:48.484381 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a5f4e0-225f-4290-b12d-a945b6a8af57 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:48.505415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adb659d-c626-41ba-b200-648a522dc0ba None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:48.543845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b71fb6e-71fe-4b5b-ab88-c2abf8690eed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:48.544801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b71fb6e-71fe-4b5b-ab88-c2abf8690eed None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:48.545230 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b71fb6e-71fe-4b5b-ab88-c2abf8690eed None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:48.545643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b71fb6e-71fe-4b5b-ab88-c2abf8690eed None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:48.546673 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b71fb6e-71fe-4b5b-ab88-c2abf8690eed None admin] RBAC: Adding query filter params (name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:48.547138 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b71fb6e-71fe-4b5b-ab88-c2abf8690eed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:48.549333 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b71fb6e-71fe-4b5b-ab88-c2abf8690eed None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:48.566267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 249/502] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:01:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:48.566267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badaf1d4-c07e-420c-8f43-714fa9d9381b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:48.566267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badaf1d4-c07e-420c-8f43-714fa9d9381b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:48.566267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badaf1d4-c07e-420c-8f43-714fa9d9381b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:48.580750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:48.587340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-badaf1d4-c07e-420c-8f43-714fa9d9381b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:48.588572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-badaf1d4-c07e-420c-8f43-714fa9d9381b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:48.589662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a5f4e0-225f-4290-b12d-a945b6a8af57 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:48.622309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7641b3-3b65-4cd5-ab9b-34980f4274f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:48.622928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7641b3-3b65-4cd5-ab9b-34980f4274f7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:48.623203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7641b3-3b65-4cd5-ab9b-34980f4274f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:48.623473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7641b3-3b65-4cd5-ab9b-34980f4274f7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:48.633545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7641b3-3b65-4cd5-ab9b-34980f4274f7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:48.634988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7641b3-3b65-4cd5-ab9b-34980f4274f7 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-cf7641b3-3b65-4cd5-ab9b-34980f4274f7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:48.648482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:48.649348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:48.650179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:48.650179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:48.650179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:48.650179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 254/503] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:48.650386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7641b3-3b65-4cd5-ab9b-34980f4274f7 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:48.665733 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 250/504] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:48.666566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badaf1d4-c07e-420c-8f43-714fa9d9381b None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:48.714607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:48.755307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7641b3-3b65-4cd5-ab9b-34980f4274f7 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:48.768876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badaf1d4-c07e-420c-8f43-714fa9d9381b None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:48.790173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd50951a-bd37-4789-b5f1-447797fc9093 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:48.791181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd50951a-bd37-4789-b5f1-447797fc9093 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:48.791471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd50951a-bd37-4789-b5f1-447797fc9093 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:48.791743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd50951a-bd37-4789-b5f1-447797fc9093 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:48.804163 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091498a4-9120-4770-95c7-c3a7b127252b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:48.804801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091498a4-9120-4770-95c7-c3a7b127252b None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:48.805063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091498a4-9120-4770-95c7-c3a7b127252b None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:48.805331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091498a4-9120-4770-95c7-c3a7b127252b None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:48.805969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091498a4-9120-4770-95c7-c3a7b127252b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:48.806269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091498a4-9120-4770-95c7-c3a7b127252b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:48.807577 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:48.807702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091498a4-9120-4770-95c7-c3a7b127252b None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:48.819829 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 251/505] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:48.821963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091498a4-9120-4770-95c7-c3a7b127252b None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:48.868544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd50951a-bd37-4789-b5f1-447797fc9093 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:48.870319 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd50951a-bd37-4789-b5f1-447797fc9093 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:48.922929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091498a4-9120-4770-95c7-c3a7b127252b None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:48.933192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:48.961388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd05990-3ce5-4dde-9221-7b7caf5aa88f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:48.962267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd05990-3ce5-4dde-9221-7b7caf5aa88f None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:48.962551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd05990-3ce5-4dde-9221-7b7caf5aa88f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:48.962821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd05990-3ce5-4dde-9221-7b7caf5aa88f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:48.972807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd05990-3ce5-4dde-9221-7b7caf5aa88f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:48.974267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd05990-3ce5-4dde-9221-7b7caf5aa88f None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:48.988001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 255/506] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:48.989407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd50951a-bd37-4789-b5f1-447797fc9093 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-4dd05990-3ce5-4dde-9221-7b7caf5aa88f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:48.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:48.990973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:48.991820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:48.991820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:48.991820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:48.991820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 252/507] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:48.992494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd05990-3ce5-4dde-9221-7b7caf5aa88f None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:49.035317 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:49.088326 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd50951a-bd37-4789-b5f1-447797fc9093 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:49.089336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd05990-3ce5-4dde-9221-7b7caf5aa88f None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:49.120671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1c821f-3575-4dfd-a896-3a3685ff3e72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:49.121395 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1c821f-3575-4dfd-a896-3a3685ff3e72 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:49.121506 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f8e861-1943-4ccf-98d8-f3c2b6502c62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:49.121687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1c821f-3575-4dfd-a896-3a3685ff3e72 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:49.122055 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1c821f-3575-4dfd-a896-3a3685ff3e72 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:49.122264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8e861-1943-4ccf-98d8-f3c2b6502c62 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:49.122501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8e861-1943-4ccf-98d8-f3c2b6502c62 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:49.122645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1c821f-3575-4dfd-a896-3a3685ff3e72 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:49.122868 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8e861-1943-4ccf-98d8-f3c2b6502c62 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:49.123004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1c821f-3575-4dfd-a896-3a3685ff3e72 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:49.124209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1c821f-3575-4dfd-a896-3a3685ff3e72 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:49.130893 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f8e861-1943-4ccf-98d8-f3c2b6502c62 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:49.132329 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f8e861-1943-4ccf-98d8-f3c2b6502c62 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:49.136189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 256/508] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:49.137003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1c821f-3575-4dfd-a896-3a3685ff3e72 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-d5f8e861-1943-4ccf-98d8-f3c2b6502c62 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:01:49.143363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:49.144456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:49.149963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:49.149963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:49.149963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:01:49.149963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:01:49.149963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:01:49.149963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:01:49.149963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:01:49.149963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:01:49.149963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 28 12:01:49.149963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:49.149963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 253/509] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:48 2022] GET /identity/v3/users/cinder => generated 84 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:49.149963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f8e861-1943-4ccf-98d8-f3c2b6502c62 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:49.167248 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:49.232255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1c821f-3575-4dfd-a896-3a3685ff3e72 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:49.234413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:49.247191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f8e861-1943-4ccf-98d8-f3c2b6502c62 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:49.266320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62630b73-8a24-4c5b-aa7c-15e23875ae95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:49.266941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62630b73-8a24-4c5b-aa7c-15e23875ae95 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:49.267218 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62630b73-8a24-4c5b-aa7c-15e23875ae95 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:49.267488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62630b73-8a24-4c5b-aa7c-15e23875ae95 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:49.277891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62630b73-8a24-4c5b-aa7c-15e23875ae95 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:49.278867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9d3780-33e4-4f35-88a3-54e09b190434 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:49.279363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62630b73-8a24-4c5b-aa7c-15e23875ae95 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:49.279504 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9d3780-33e4-4f35-88a3-54e09b190434 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:49.279784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9d3780-33e4-4f35-88a3-54e09b190434 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:49.280056 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9d3780-33e4-4f35-88a3-54e09b190434 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:49.291514 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9d3780-33e4-4f35-88a3-54e09b190434 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:49.292995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9d3780-33e4-4f35-88a3-54e09b190434 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-62630b73-8a24-4c5b-aa7c-15e23875ae95 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:49.296788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:49.297833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:49.298781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:49.298781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:49.298781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:49.298781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 257/510] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:49.299086 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62630b73-8a24-4c5b-aa7c-15e23875ae95 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-2d9d3780-33e4-4f35-88a3-54e09b190434 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:49.312529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:49.313659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:49.314672 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:49.314672 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:49.314672 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:49.314672 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 254/511] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:49.314672 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9d3780-33e4-4f35-88a3-54e09b190434 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:49.381902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:49.401807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62630b73-8a24-4c5b-aa7c-15e23875ae95 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:49.413512 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9d3780-33e4-4f35-88a3-54e09b190434 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:49.434956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328b639d-36f0-4e00-a3b2-5505934874cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:49.435554 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328b639d-36f0-4e00-a3b2-5505934874cd None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:49.435837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328b639d-36f0-4e00-a3b2-5505934874cd None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:49.436109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328b639d-36f0-4e00-a3b2-5505934874cd None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:49.436778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328b639d-36f0-4e00-a3b2-5505934874cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:49.437082 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328b639d-36f0-4e00-a3b2-5505934874cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:49.438478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328b639d-36f0-4e00-a3b2-5505934874cd None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:49.446335 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7da684-ee9d-4409-b84d-684db931b10a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:49.446962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7da684-ee9d-4409-b84d-684db931b10a None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:49.447242 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7da684-ee9d-4409-b84d-684db931b10a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:49.447513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7da684-ee9d-4409-b84d-684db931b10a None admin] PATH_INFO: `/v3/users/glance` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:49.451054 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 258/512] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:49.452902 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328b639d-36f0-4e00-a3b2-5505934874cd None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:49.458265 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7da684-ee9d-4409-b84d-684db931b10a None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:49.459713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7da684-ee9d-4409-b84d-684db931b10a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:49.470746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-8e7da684-ee9d-4409-b84d-684db931b10a None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:01:49.471172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:49.472139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:49.473553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:49.473553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:49.473553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:01:49.473553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:01:49.473553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:01:49.473553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:01:49.473553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:01:49.473553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:01:49.473553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 28 12:01:49.473553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:49.473553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 255/513] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:49 2022] GET /identity/v3/users/glance => generated 84 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:49.473553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7da684-ee9d-4409-b84d-684db931b10a None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:49.552720 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328b639d-36f0-4e00-a3b2-5505934874cd None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:49.571589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7da684-ee9d-4409-b84d-684db931b10a None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:49.589579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2265a28-5583-4a46-b43b-6eb52b6f1a22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:49.590277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2265a28-5583-4a46-b43b-6eb52b6f1a22 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:49.590565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2265a28-5583-4a46-b43b-6eb52b6f1a22 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:49.590841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2265a28-5583-4a46-b43b-6eb52b6f1a22 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:49.591573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2265a28-5583-4a46-b43b-6eb52b6f1a22 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:49.591871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2265a28-5583-4a46-b43b-6eb52b6f1a22 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:49.593334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2265a28-5583-4a46-b43b-6eb52b6f1a22 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:49.607100 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:49.614307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07583da5-068f-4de8-8f4a-6ce58cede338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:49.614915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07583da5-068f-4de8-8f4a-6ce58cede338 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:49.615197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07583da5-068f-4de8-8f4a-6ce58cede338 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:49.615474 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07583da5-068f-4de8-8f4a-6ce58cede338 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:49.616123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07583da5-068f-4de8-8f4a-6ce58cede338 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:49.616464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07583da5-068f-4de8-8f4a-6ce58cede338 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:49.617844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07583da5-068f-4de8-8f4a-6ce58cede338 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:49.629825 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 259/514] 10.222.0.26 () {56 vars in 1137 bytes} [Wed Sep 28 12:01:49 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:49.634660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 256/515] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:49.634818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2265a28-5583-4a46-b43b-6eb52b6f1a22 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:49.634907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07583da5-068f-4de8-8f4a-6ce58cede338 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:49.705259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:49.731196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2265a28-5583-4a46-b43b-6eb52b6f1a22 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:49.731349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07583da5-068f-4de8-8f4a-6ce58cede338 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:49.762971 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c847f48f-5665-4c8d-b16a-863fa09c1ac9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:49.763433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1117a56c-f1d3-4765-8e06-5c74d8de10b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:49.763619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c847f48f-5665-4c8d-b16a-863fa09c1ac9 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:49.763898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c847f48f-5665-4c8d-b16a-863fa09c1ac9 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:49.764040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1117a56c-f1d3-4765-8e06-5c74d8de10b7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:49.764190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c847f48f-5665-4c8d-b16a-863fa09c1ac9 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:49.764301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1117a56c-f1d3-4765-8e06-5c74d8de10b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:49.764675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1117a56c-f1d3-4765-8e06-5c74d8de10b7 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:49.765320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1117a56c-f1d3-4765-8e06-5c74d8de10b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:49.765625 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1117a56c-f1d3-4765-8e06-5c74d8de10b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:49.767002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1117a56c-f1d3-4765-8e06-5c74d8de10b7 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:49.771465 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c847f48f-5665-4c8d-b16a-863fa09c1ac9 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:49.772920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c847f48f-5665-4c8d-b16a-863fa09c1ac9 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:49.778353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 260/516] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:49.779142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1117a56c-f1d3-4765-8e06-5c74d8de10b7 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-c847f48f-5665-4c8d-b16a-863fa09c1ac9 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:01:49.786303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:49.787373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:49.788516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:49.788516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:49.788516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:01:49.788516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:01:49.788516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:01:49.788516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:01:49.788516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:01:49.788516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:01:49.788516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 28 12:01:49.788516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:49.788516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 257/517] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:01:49 2022] GET /identity/v3/users/neutron => generated 85 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:49.789399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c847f48f-5665-4c8d-b16a-863fa09c1ac9 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:49.836909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:49.879073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1117a56c-f1d3-4765-8e06-5c74d8de10b7 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:49.888324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c847f48f-5665-4c8d-b16a-863fa09c1ac9 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:49.912715 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6356ace2-90bf-455d-bb4a-a52536054073 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:49.913316 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6356ace2-90bf-455d-bb4a-a52536054073 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:49.913595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6356ace2-90bf-455d-bb4a-a52536054073 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:49.913867 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6356ace2-90bf-455d-bb4a-a52536054073 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:49.914590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6356ace2-90bf-455d-bb4a-a52536054073 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:49.914894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6356ace2-90bf-455d-bb4a-a52536054073 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:49.916354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6356ace2-90bf-455d-bb4a-a52536054073 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:49.917926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:49.922134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3727629b-505a-44bc-8b29-1fb4fd7e593c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:49.922732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3727629b-505a-44bc-8b29-1fb4fd7e593c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:49.923020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3727629b-505a-44bc-8b29-1fb4fd7e593c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:49.923292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3727629b-505a-44bc-8b29-1fb4fd7e593c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:49.936834 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 261/518] 10.222.0.26 () {56 vars in 1137 bytes} [Wed Sep 28 12:01:49 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:49.937921 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6356ace2-90bf-455d-bb4a-a52536054073 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:49.996406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3727629b-505a-44bc-8b29-1fb4fd7e593c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:49.997869 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3727629b-505a-44bc-8b29-1fb4fd7e593c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:50.043319 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6356ace2-90bf-455d-bb4a-a52536054073 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:50.045456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:50.077645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c65735-c122-4da9-8559-8d61279d9099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:50.078247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c65735-c122-4da9-8559-8d61279d9099 None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:50.078528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c65735-c122-4da9-8559-8d61279d9099 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:50.078801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c65735-c122-4da9-8559-8d61279d9099 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:50.079448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c65735-c122-4da9-8559-8d61279d9099 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:50.080924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c65735-c122-4da9-8559-8d61279d9099 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:50.097283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 258/519] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:50.098383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3727629b-505a-44bc-8b29-1fb4fd7e593c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:50.106453 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 262/520] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:01:49 2022] POST /identity/v3/users => generated 247 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:01:50.107266 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c65735-c122-4da9-8559-8d61279d9099 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:50.174799 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:50.211623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3727629b-505a-44bc-8b29-1fb4fd7e593c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:50.234407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c65735-c122-4da9-8559-8d61279d9099 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:50.248361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8251249-9009-41e0-aad2-fd8ff4a1b4e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:50.249007 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8251249-9009-41e0-aad2-fd8ff4a1b4e5 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:50.249288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8251249-9009-41e0-aad2-fd8ff4a1b4e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:50.249563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8251249-9009-41e0-aad2-fd8ff4a1b4e5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:50.257308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8251249-9009-41e0-aad2-fd8ff4a1b4e5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:50.258775 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8251249-9009-41e0-aad2-fd8ff4a1b4e5 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-b8251249-9009-41e0-aad2-fd8ff4a1b4e5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:01:50.271753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:50.272719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:50.273867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:01:50.273867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:01:50.273867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:01:50.273867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:01:50.273867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:01:50.273867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:01:50.273867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:50.273867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:50.276814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b31b0dc-d284-45b3-add6-c9fcb7bc2c19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:50.277829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b31b0dc-d284-45b3-add6-c9fcb7bc2c19 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:50.277944 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 259/521] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:50.278311 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b31b0dc-d284-45b3-add6-c9fcb7bc2c19 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:50.278814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b31b0dc-d284-45b3-add6-c9fcb7bc2c19 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:50.278941 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8251249-9009-41e0-aad2-fd8ff4a1b4e5 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:50.280787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b31b0dc-d284-45b3-add6-c9fcb7bc2c19 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:50.280897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b31b0dc-d284-45b3-add6-c9fcb7bc2c19 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:50.282895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b31b0dc-d284-45b3-add6-c9fcb7bc2c19 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:50.294685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:50.308360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 263/522] 10.222.0.26 () {56 vars in 1139 bytes} [Wed Sep 28 12:01:50 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:50.309213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b31b0dc-d284-45b3-add6-c9fcb7bc2c19 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:50.401582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8251249-9009-41e0-aad2-fd8ff4a1b4e5 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:50.420699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b31b0dc-d284-45b3-add6-c9fcb7bc2c19 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:50.429171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:50.437957 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d732b3ff-60fc-4694-9f67-bfbaed7631a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:50.438623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d732b3ff-60fc-4694-9f67-bfbaed7631a9 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:50.438902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d732b3ff-60fc-4694-9f67-bfbaed7631a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:50.439185 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d732b3ff-60fc-4694-9f67-bfbaed7631a9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:50.457707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5afe3a-1cc8-4cc9-a7e4-c8893de7f91d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:50.458435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5afe3a-1cc8-4cc9-a7e4-c8893de7f91d None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:50.458711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5afe3a-1cc8-4cc9-a7e4-c8893de7f91d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:50.459095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5afe3a-1cc8-4cc9-a7e4-c8893de7f91d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:50.470717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5afe3a-1cc8-4cc9-a7e4-c8893de7f91d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:50.472432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5afe3a-1cc8-4cc9-a7e4-c8893de7f91d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-5c5afe3a-1cc8-4cc9-a7e4-c8893de7f91d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:50.492667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:50.493787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:50.495372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:50.495372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:50.495372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:50.496468 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 264/523] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:50.498105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5afe3a-1cc8-4cc9-a7e4-c8893de7f91d None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:50.517986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d732b3ff-60fc-4694-9f67-bfbaed7631a9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:50.519436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d732b3ff-60fc-4694-9f67-bfbaed7631a9 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:50.531344 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:50.610984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5afe3a-1cc8-4cc9-a7e4-c8893de7f91d None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:50.629017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 260/524] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:50.630127 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d732b3ff-60fc-4694-9f67-bfbaed7631a9 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:50.655951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8e588e-23f7-4bfd-aca7-9ce78e72b606 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:50.656707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8e588e-23f7-4bfd-aca7-9ce78e72b606 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:50.657056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8e588e-23f7-4bfd-aca7-9ce78e72b606 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:50.657401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8e588e-23f7-4bfd-aca7-9ce78e72b606 None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:50.658170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8e588e-23f7-4bfd-aca7-9ce78e72b606 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:50.658545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8e588e-23f7-4bfd-aca7-9ce78e72b606 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:50.659990 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8e588e-23f7-4bfd-aca7-9ce78e72b606 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:50.682921 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 265/525] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:01:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:50.683800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8e588e-23f7-4bfd-aca7-9ce78e72b606 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:50.702497 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:50.735025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d732b3ff-60fc-4694-9f67-bfbaed7631a9 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:50.745922 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:50.777641 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcae661-23b6-42d7-82bb-24658a2eb981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:50.778565 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcae661-23b6-42d7-82bb-24658a2eb981 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:50.778988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcae661-23b6-42d7-82bb-24658a2eb981 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:50.779394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcae661-23b6-42d7-82bb-24658a2eb981 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:50.786477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8e588e-23f7-4bfd-aca7-9ce78e72b606 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:50.819665 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf77965-cf88-4d18-b2c3-51f40a9c55df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:50.820282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf77965-cf88-4d18-b2c3-51f40a9c55df None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:50.820590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf77965-cf88-4d18-b2c3-51f40a9c55df None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:50.820861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf77965-cf88-4d18-b2c3-51f40a9c55df None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:50.821502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf77965-cf88-4d18-b2c3-51f40a9c55df None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:50.821803 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf77965-cf88-4d18-b2c3-51f40a9c55df None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:50.823186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf77965-cf88-4d18-b2c3-51f40a9c55df None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:50.839625 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 266/526] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:50.839625 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf77965-cf88-4d18-b2c3-51f40a9c55df None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:50.851431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcae661-23b6-42d7-82bb-24658a2eb981 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:50.852931 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcae661-23b6-42d7-82bb-24658a2eb981 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:50.924368 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:50.954354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf77965-cf88-4d18-b2c3-51f40a9c55df None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:50.968346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 261/527] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:01:50.968346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcae661-23b6-42d7-82bb-24658a2eb981 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:50.980592 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:50.999011 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d820e4fd-f04a-4dae-9c01-ccbda88c0781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:50.999635 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d820e4fd-f04a-4dae-9c01-ccbda88c0781 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:50.999910 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d820e4fd-f04a-4dae-9c01-ccbda88c0781 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:51.000178 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d820e4fd-f04a-4dae-9c01-ccbda88c0781 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:51.010714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d820e4fd-f04a-4dae-9c01-ccbda88c0781 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:51.012180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d820e4fd-f04a-4dae-9c01-ccbda88c0781 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-d820e4fd-f04a-4dae-9c01-ccbda88c0781 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:51.026323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:51.027332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:51.028201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:51.028201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:51.028201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:51.031913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 267/528] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:51.032624 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d820e4fd-f04a-4dae-9c01-ccbda88c0781 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:51.077482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcae661-23b6-42d7-82bb-24658a2eb981 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:51.111138 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171f0a28-9580-4926-82cf-7e0d777a9286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:51.111733 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171f0a28-9580-4926-82cf-7e0d777a9286 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:51.112009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171f0a28-9580-4926-82cf-7e0d777a9286 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:51.112280 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171f0a28-9580-4926-82cf-7e0d777a9286 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:51.113027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171f0a28-9580-4926-82cf-7e0d777a9286 None admin] RBAC: Adding query filter params (role.id=ed4d262295434a72bb0a8b9a7ff12e71, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=8bf969348e7a4158acf8fb789430b0e6) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:51.113332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171f0a28-9580-4926-82cf-7e0d777a9286 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:51.115399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171f0a28-9580-4926-82cf-7e0d777a9286 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:51.120959 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 262/529] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:50 2022] GET /identity/v3/role_assignments?user.id=8bf969348e7a4158acf8fb789430b0e6&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=ed4d262295434a72bb0a8b9a7ff12e71 => generated 259 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:51.121738 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171f0a28-9580-4926-82cf-7e0d777a9286 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:51.165564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d820e4fd-f04a-4dae-9c01-ccbda88c0781 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:51.204496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:51.206098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:51.213412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b8705e-efff-4325-b0b5-c7be2d23fc7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:51.214754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b8705e-efff-4325-b0b5-c7be2d23fc7b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:51.215637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b8705e-efff-4325-b0b5-c7be2d23fc7b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:51.216121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b8705e-efff-4325-b0b5-c7be2d23fc7b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:51.225159 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b8705e-efff-4325-b0b5-c7be2d23fc7b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:51.227217 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b8705e-efff-4325-b0b5-c7be2d23fc7b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:51.237856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171f0a28-9580-4926-82cf-7e0d777a9286 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-93b8705e-efff-4325-b0b5-c7be2d23fc7b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:01:51.240572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:51.241422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:51.242264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:01:51.242264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:01:51.242264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:01:51.242264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:01:51.242264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:01:51.242264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:01:51.242264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:51.242264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:51.242264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 268/530] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:51.242712 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b8705e-efff-4325-b0b5-c7be2d23fc7b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:51.281987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562244a4-c21f-411e-b701-9692d2cb960f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:51.283016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562244a4-c21f-411e-b701-9692d2cb960f None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:51.283434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562244a4-c21f-411e-b701-9692d2cb960f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:51.283871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562244a4-c21f-411e-b701-9692d2cb960f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:51.296745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562244a4-c21f-411e-b701-9692d2cb960f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:51.298456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562244a4-c21f-411e-b701-9692d2cb960f None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-562244a4-c21f-411e-b701-9692d2cb960f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:51.316387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:51.317580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:51.318606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:51.318606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:51.318606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:51.324871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 263/531] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:51.325822 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562244a4-c21f-411e-b701-9692d2cb960f None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:51.381012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b8705e-efff-4325-b0b5-c7be2d23fc7b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:51.421587 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae35ffcf-581d-4907-a6f2-ac040bc459d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:51.422194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae35ffcf-581d-4907-a6f2-ac040bc459d2 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:51.422466 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae35ffcf-581d-4907-a6f2-ac040bc459d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:51.422728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae35ffcf-581d-4907-a6f2-ac040bc459d2 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:51.423365 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae35ffcf-581d-4907-a6f2-ac040bc459d2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:51.423667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae35ffcf-581d-4907-a6f2-ac040bc459d2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:51.425078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae35ffcf-581d-4907-a6f2-ac040bc459d2 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:51.443417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 269/532] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:51.444250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae35ffcf-581d-4907-a6f2-ac040bc459d2 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:51.460629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562244a4-c21f-411e-b701-9692d2cb960f None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:51.498386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:51.515961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca4f50e-be3e-4f47-8260-0485a53e3833 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:51.517403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca4f50e-be3e-4f47-8260-0485a53e3833 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:51.518150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca4f50e-be3e-4f47-8260-0485a53e3833 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:51.519301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca4f50e-be3e-4f47-8260-0485a53e3833 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:51.520410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:51.520878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca4f50e-be3e-4f47-8260-0485a53e3833 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:51.521753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca4f50e-be3e-4f47-8260-0485a53e3833 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:51.522963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca4f50e-be3e-4f47-8260-0485a53e3833 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:51.548361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 264/533] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:01:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:51.548361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca4f50e-be3e-4f47-8260-0485a53e3833 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:51.585895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae35ffcf-581d-4907-a6f2-ac040bc459d2 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:51.620934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f9e002-a672-4aad-b441-3e52d042a5f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:51.621902 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f9e002-a672-4aad-b441-3e52d042a5f8 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:51.622404 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f9e002-a672-4aad-b441-3e52d042a5f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:51.622819 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f9e002-a672-4aad-b441-3e52d042a5f8 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:51.623673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f9e002-a672-4aad-b441-3e52d042a5f8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:51.624121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f9e002-a672-4aad-b441-3e52d042a5f8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:51.625688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f9e002-a672-4aad-b441-3e52d042a5f8 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:51.638176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 270/534] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:51.639240 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f9e002-a672-4aad-b441-3e52d042a5f8 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:51.675024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca4f50e-be3e-4f47-8260-0485a53e3833 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:51.718700 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37381c26-06b1-4920-84a1-61818b675aa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:51.719251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37381c26-06b1-4920-84a1-61818b675aa2 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:51.719530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37381c26-06b1-4920-84a1-61818b675aa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:51.719802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37381c26-06b1-4920-84a1-61818b675aa2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:51.729100 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37381c26-06b1-4920-84a1-61818b675aa2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:51.730864 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37381c26-06b1-4920-84a1-61818b675aa2 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-37381c26-06b1-4920-84a1-61818b675aa2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:01:51.745824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:51.746708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:51.747553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:01:51.747553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:01:51.747553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:01:51.747553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:01:51.747553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:01:51.747553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:01:51.747553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:51.747553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:51.749155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f9e002-a672-4aad-b441-3e52d042a5f8 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:51.750179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 265/535] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:51.751318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37381c26-06b1-4920-84a1-61818b675aa2 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:51.761678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:51.794129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c2c041-c8a4-427d-b424-a8df62b0fe4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:51.794763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c2c041-c8a4-427d-b424-a8df62b0fe4f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:51.795041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c2c041-c8a4-427d-b424-a8df62b0fe4f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:51.795318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c2c041-c8a4-427d-b424-a8df62b0fe4f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:51.796047 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c2c041-c8a4-427d-b424-a8df62b0fe4f None admin] RBAC: Adding query filter params (role.id=ed4d262295434a72bb0a8b9a7ff12e71, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=2ba80861b08940d398a61879bb70060a) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:51.796375 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c2c041-c8a4-427d-b424-a8df62b0fe4f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:51.798366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c2c041-c8a4-427d-b424-a8df62b0fe4f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:51.807543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:51.812459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 271/536] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:51 2022] GET /identity/v3/role_assignments?user.id=2ba80861b08940d398a61879bb70060a&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=ed4d262295434a72bb0a8b9a7ff12e71 => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:51.813747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c2c041-c8a4-427d-b424-a8df62b0fe4f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:51.922218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37381c26-06b1-4920-84a1-61818b675aa2 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:51.996908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f1b6cf-e734-45d0-b96d-d69ebaa8fa0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:51.998178 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f1b6cf-e734-45d0-b96d-d69ebaa8fa0f None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:51.999843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f1b6cf-e734-45d0-b96d-d69ebaa8fa0f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:52.000566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f1b6cf-e734-45d0-b96d-d69ebaa8fa0f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:52.017053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f1b6cf-e734-45d0-b96d-d69ebaa8fa0f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:52.018557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f1b6cf-e734-45d0-b96d-d69ebaa8fa0f None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:52.019632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c2c041-c8a4-427d-b424-a8df62b0fe4f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-79f1b6cf-e734-45d0-b96d-d69ebaa8fa0f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:01:52.039358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:52.040244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:52.041098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:01:52.041098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:01:52.041098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:01:52.041098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:01:52.041098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:01:52.041098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:01:52.041098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:52.041098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:52.045664 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 266/537] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:52.051480 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f1b6cf-e734-45d0-b96d-d69ebaa8fa0f None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:52.088015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b7a8d0-5b8e-4e92-a87f-0c34752a95c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:52.089045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b7a8d0-5b8e-4e92-a87f-0c34752a95c3 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:52.091676 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b7a8d0-5b8e-4e92-a87f-0c34752a95c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:52.092156 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b7a8d0-5b8e-4e92-a87f-0c34752a95c3 None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:52.093100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b7a8d0-5b8e-4e92-a87f-0c34752a95c3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:52.093605 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b7a8d0-5b8e-4e92-a87f-0c34752a95c3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:52.097063 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b7a8d0-5b8e-4e92-a87f-0c34752a95c3 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:52.128041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 272/538] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:01:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:52.132680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b7a8d0-5b8e-4e92-a87f-0c34752a95c3 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:52.204817 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:52.248025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f1b6cf-e734-45d0-b96d-d69ebaa8fa0f None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:52.260597 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:52.314390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b7a8d0-5b8e-4e92-a87f-0c34752a95c3 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:52.339831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f19e02-d2e8-445a-aa41-802b0842a6c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:52.340489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f19e02-d2e8-445a-aa41-802b0842a6c6 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:52.340768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f19e02-d2e8-445a-aa41-802b0842a6c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:52.341040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f19e02-d2e8-445a-aa41-802b0842a6c6 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:52.341737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f19e02-d2e8-445a-aa41-802b0842a6c6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:52.342042 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f19e02-d2e8-445a-aa41-802b0842a6c6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:52.344343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f19e02-d2e8-445a-aa41-802b0842a6c6 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:52.369987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 267/539] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:01:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:52.374730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f19e02-d2e8-445a-aa41-802b0842a6c6 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:52.385009 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10a771b-1b95-43d4-96b4-9142e6afd5af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:52.385619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a771b-1b95-43d4-96b4-9142e6afd5af None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:52.385897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a771b-1b95-43d4-96b4-9142e6afd5af None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:52.386163 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a771b-1b95-43d4-96b4-9142e6afd5af None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:52.386896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10a771b-1b95-43d4-96b4-9142e6afd5af None admin] RBAC: Adding query filter params (role.id=ed4d262295434a72bb0a8b9a7ff12e71, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=815504693bcc40b6aef06aacb2a9a35b) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:52.387204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10a771b-1b95-43d4-96b4-9142e6afd5af None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:52.388686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10a771b-1b95-43d4-96b4-9142e6afd5af None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:52.398206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 273/540] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:52 2022] GET /identity/v3/role_assignments?user.id=815504693bcc40b6aef06aacb2a9a35b&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=ed4d262295434a72bb0a8b9a7ff12e71 => generated 259 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:52.399774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cacfa8-5021-4db3-8699-71fb91dbe0f3 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:52.400058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cacfa8-5021-4db3-8699-71fb91dbe0f3 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:52.400337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cacfa8-5021-4db3-8699-71fb91dbe0f3 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:52.401601 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 274/541] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:52.407661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3157357f-9ec5-4e67-9e90-1a0fbdb5fd52 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:52.407951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3157357f-9ec5-4e67-9e90-1a0fbdb5fd52 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:52.408223 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3157357f-9ec5-4e67-9e90-1a0fbdb5fd52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:52.429835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3157357f-9ec5-4e67-9e90-1a0fbdb5fd52 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:52.431493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3157357f-9ec5-4e67-9e90-1a0fbdb5fd52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:52.570205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f19e02-d2e8-445a-aa41-802b0842a6c6 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:52.575748 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 275/542] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:52.625357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd053763-b380-4356-82c6-a8e29483f205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:52.626213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd053763-b380-4356-82c6-a8e29483f205 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:52.626637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd053763-b380-4356-82c6-a8e29483f205 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:52.627070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd053763-b380-4356-82c6-a8e29483f205 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:52.627955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd053763-b380-4356-82c6-a8e29483f205 None admin] RBAC: Adding query filter params (role.id=ed4d262295434a72bb0a8b9a7ff12e71, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=0e429f061bd04c28b0c60c593b64fb8e) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:52.628474 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd053763-b380-4356-82c6-a8e29483f205 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:52.630069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd053763-b380-4356-82c6-a8e29483f205 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:52.640529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 268/543] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:52 2022] GET /identity/v3/role_assignments?user.id=0e429f061bd04c28b0c60c593b64fb8e&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=ed4d262295434a72bb0a8b9a7ff12e71 => generated 259 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:52.968235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3157357f-9ec5-4e67-9e90-1a0fbdb5fd52 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:53.008782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:53.107167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3157357f-9ec5-4e67-9e90-1a0fbdb5fd52 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:53.152986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c6ab86-c3d5-4322-97eb-92b6422d4c8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:53.153622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c6ab86-c3d5-4322-97eb-92b6422d4c8e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:53.153897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c6ab86-c3d5-4322-97eb-92b6422d4c8e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:53.154164 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c6ab86-c3d5-4322-97eb-92b6422d4c8e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:53.170895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c6ab86-c3d5-4322-97eb-92b6422d4c8e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-b0c6ab86-c3d5-4322-97eb-92b6422d4c8e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:53.175717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:53.176651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:53.177441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:53.177441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:53.177441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:53.177441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:53.177441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:53.177441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:53.177441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:53.177441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:53.177441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:53.177441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:53.177441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:53.177441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:53.181671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 276/544] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:01:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:53.183066 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd053763-b380-4356-82c6-a8e29483f205 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:53.214881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:53.293872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd053763-b380-4356-82c6-a8e29483f205 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:53.339780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342001f9-1f5d-4574-925c-0f4ed148ecd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:53.340424 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342001f9-1f5d-4574-925c-0f4ed148ecd0 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:53.340806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342001f9-1f5d-4574-925c-0f4ed148ecd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:53.341088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342001f9-1f5d-4574-925c-0f4ed148ecd0 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:53.341765 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342001f9-1f5d-4574-925c-0f4ed148ecd0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:53.342068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342001f9-1f5d-4574-925c-0f4ed148ecd0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:53.343458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342001f9-1f5d-4574-925c-0f4ed148ecd0 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:53.355634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 269/545] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:01:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:53.361470 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c6ab86-c3d5-4322-97eb-92b6422d4c8e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:53.475012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:53.487947 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c6ab86-c3d5-4322-97eb-92b6422d4c8e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:53.525222 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3d33dc-dc25-4b98-afe1-4129d1aa0b47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:53.525852 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3d33dc-dc25-4b98-afe1-4129d1aa0b47 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:53.526131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3d33dc-dc25-4b98-afe1-4129d1aa0b47 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:53.526403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3d33dc-dc25-4b98-afe1-4129d1aa0b47 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:53.537733 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3d33dc-dc25-4b98-afe1-4129d1aa0b47 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:53.539203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3d33dc-dc25-4b98-afe1-4129d1aa0b47 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:53.544740 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba0921e-9065-4ff0-8c29-ee2b2e34c8e6 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:53.544967 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba0921e-9065-4ff0-8c29-ee2b2e34c8e6 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:53.545239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba0921e-9065-4ff0-8c29-ee2b2e34c8e6 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:53.549989 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 270/546] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:01:53.552533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3378d016-a7f9-41cb-8dbf-b69f484efad1 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:53.552821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3378d016-a7f9-41cb-8dbf-b69f484efad1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:53.553098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3378d016-a7f9-41cb-8dbf-b69f484efad1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-2c3d33dc-dc25-4b98-afe1-4129d1aa0b47 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:53.558975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:53.559893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:53.560755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:53.560755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:53.560755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:53.564334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 277/547] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:53.566883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3d33dc-dc25-4b98-afe1-4129d1aa0b47 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:53.581277 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-3378d016-a7f9-41cb-8dbf-b69f484efad1 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:53.584788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-3378d016-a7f9-41cb-8dbf-b69f484efad1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:53.717918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 271/548] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:53.729717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3d33dc-dc25-4b98-afe1-4129d1aa0b47 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:53.778943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5e0e4d-53be-4028-9144-5d7e97354222 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:53.779947 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5e0e4d-53be-4028-9144-5d7e97354222 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:53.780513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5e0e4d-53be-4028-9144-5d7e97354222 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:53.780992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5e0e4d-53be-4028-9144-5d7e97354222 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:53.781881 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5e0e4d-53be-4028-9144-5d7e97354222 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:53.782385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5e0e4d-53be-4028-9144-5d7e97354222 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:53.784018 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5e0e4d-53be-4028-9144-5d7e97354222 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:53.798660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:53.803923 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 278/549] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:53.809204 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3378d016-a7f9-41cb-8dbf-b69f484efad1 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:53.851289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:53.952772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3378d016-a7f9-41cb-8dbf-b69f484efad1 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:53.998845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b75417e-8990-41b1-af93-ef036cd69db0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:53.999476 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b75417e-8990-41b1-af93-ef036cd69db0 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:53.999756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b75417e-8990-41b1-af93-ef036cd69db0 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:54.000028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b75417e-8990-41b1-af93-ef036cd69db0 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:54.009277 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b75417e-8990-41b1-af93-ef036cd69db0 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:54.010703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b75417e-8990-41b1-af93-ef036cd69db0 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-6b75417e-8990-41b1-af93-ef036cd69db0 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:01:54.022152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:54.023629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:54.024647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:54.024647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:54.024647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:01:54.024647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:01:54.024647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:01:54.024647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:01:54.024647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:01:54.024647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:01:54.024647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 28 12:01:54.024647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:54.027873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 272/550] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:01:53 2022] GET /identity/v3/users/swift => generated 83 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:54.029251 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5e0e4d-53be-4028-9144-5d7e97354222 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:54.134599 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b75417e-8990-41b1-af93-ef036cd69db0 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:54.140917 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5e0e4d-53be-4028-9144-5d7e97354222 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:54.207923 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1842e609-f50e-4965-bca5-eb58733f83b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:54.208555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1842e609-f50e-4965-bca5-eb58733f83b3 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:54.208833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1842e609-f50e-4965-bca5-eb58733f83b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:54.209101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1842e609-f50e-4965-bca5-eb58733f83b3 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:54.209830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1842e609-f50e-4965-bca5-eb58733f83b3 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:54.210133 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1842e609-f50e-4965-bca5-eb58733f83b3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:54.211577 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1842e609-f50e-4965-bca5-eb58733f83b3 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:54.239968 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 279/551] 10.222.0.26 () {56 vars in 1135 bytes} [Wed Sep 28 12:01:54 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:54.246508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1842e609-f50e-4965-bca5-eb58733f83b3 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:54.262433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:54.294843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b75417e-8990-41b1-af93-ef036cd69db0 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:54.355568 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:54.357610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df8c14e-957a-4ea6-a6dd-8bd912b08680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:54.358442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df8c14e-957a-4ea6-a6dd-8bd912b08680 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:54.358872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df8c14e-957a-4ea6-a6dd-8bd912b08680 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:54.359396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df8c14e-957a-4ea6-a6dd-8bd912b08680 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:54.371999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df8c14e-957a-4ea6-a6dd-8bd912b08680 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-6df8c14e-957a-4ea6-a6dd-8bd912b08680 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:54.378003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:54.379153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:54.380490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:54.380490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:54.380490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:54.380490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:54.380490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:54.380490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:54.380490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:54.380490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:54.380490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:54.380490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:54.380490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:54.380490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:54.382927 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 273/552] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:01:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:54.388415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df8c14e-957a-4ea6-a6dd-8bd912b08680 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:54.433923 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1842e609-f50e-4965-bca5-eb58733f83b3 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:54.500624 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887bee11-3526-45f2-9656-c9d3446296c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:54.500624 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887bee11-3526-45f2-9656-c9d3446296c5 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:54.500624 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887bee11-3526-45f2-9656-c9d3446296c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:54.500624 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887bee11-3526-45f2-9656-c9d3446296c5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:54.572581 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df8c14e-957a-4ea6-a6dd-8bd912b08680 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:54.602912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887bee11-3526-45f2-9656-c9d3446296c5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:54.604619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887bee11-3526-45f2-9656-c9d3446296c5 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:54.607249 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7bcbc0-a828-4eef-8bb0-f11d13200cce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:54.607859 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7bcbc0-a828-4eef-8bb0-f11d13200cce None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:54.608134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7bcbc0-a828-4eef-8bb0-f11d13200cce None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:54.608430 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7bcbc0-a828-4eef-8bb0-f11d13200cce None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:54.609094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7bcbc0-a828-4eef-8bb0-f11d13200cce None admin] RBAC: Adding query filter params (name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:54.609390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7bcbc0-a828-4eef-8bb0-f11d13200cce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:54.610721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7bcbc0-a828-4eef-8bb0-f11d13200cce None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:54.619168 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 274/553] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:01:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:54.624428 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b59f4d7-b6a6-4127-a7cc-9b4c2a8b6323 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:54.624749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b59f4d7-b6a6-4127-a7cc-9b4c2a8b6323 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:54.625088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b59f4d7-b6a6-4127-a7cc-9b4c2a8b6323 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:54.631013 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 275/554] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:54 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:54.631013 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b59f4d7-b6a6-4127-a7cc-9b4c2a8b6323 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:54.632004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:54.681120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:54.737950 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b59f4d7-b6a6-4127-a7cc-9b4c2a8b6323 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:54.747694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 280/555] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:01:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 28 12:01:54.750186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c6f336-59c8-4897-9a01-8773657a717d None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:54.750739 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c6f336-59c8-4897-9a01-8773657a717d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:54.751652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c6f336-59c8-4897-9a01-8773657a717d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:54.775591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1c3cf3-ebdb-4f73-a334-0c1e6eb30730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:54.776213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1c3cf3-ebdb-4f73-a334-0c1e6eb30730 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:54.776518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1c3cf3-ebdb-4f73-a334-0c1e6eb30730 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:54.776794 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1c3cf3-ebdb-4f73-a334-0c1e6eb30730 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:54.777277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-31c6f336-59c8-4897-9a01-8773657a717d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:54.779094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-31c6f336-59c8-4897-9a01-8773657a717d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:54.788771 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1c3cf3-ebdb-4f73-a334-0c1e6eb30730 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:54.790219 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1c3cf3-ebdb-4f73-a334-0c1e6eb30730 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-be1c3cf3-ebdb-4f73-a334-0c1e6eb30730 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:54.808562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:54.809511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:54.810358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:54.810358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:54.810358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:54.810358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 276/556] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:54.810994 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1c3cf3-ebdb-4f73-a334-0c1e6eb30730 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:54.855585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 281/557] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:54.856845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c6f336-59c8-4897-9a01-8773657a717d None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:54.891549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:54.956429 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1c3cf3-ebdb-4f73-a334-0c1e6eb30730 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:54.988356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:54.994947 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2468123e-a688-4a26-8368-e0e964bc906c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:54.998417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2468123e-a688-4a26-8368-e0e964bc906c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:54.998417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2468123e-a688-4a26-8368-e0e964bc906c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:54.998417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2468123e-a688-4a26-8368-e0e964bc906c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.009294 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2468123e-a688-4a26-8368-e0e964bc906c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:55.009854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c6f336-59c8-4897-9a01-8773657a717d None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:55.010929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2468123e-a688-4a26-8368-e0e964bc906c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-2468123e-a688-4a26-8368-e0e964bc906c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:55.027737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:55.029216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:55.030123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:55.030123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:55.030123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:55.033616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 277/558] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:55.034790 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2468123e-a688-4a26-8368-e0e964bc906c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:55.079978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dc1a26-a948-4bd3-b4e3-8e251e1824df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:55.081023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dc1a26-a948-4bd3-b4e3-8e251e1824df None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:55.081536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dc1a26-a948-4bd3-b4e3-8e251e1824df None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:55.082014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dc1a26-a948-4bd3-b4e3-8e251e1824df None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.082983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dc1a26-a948-4bd3-b4e3-8e251e1824df None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:55.083506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dc1a26-a948-4bd3-b4e3-8e251e1824df None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:55.085194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dc1a26-a948-4bd3-b4e3-8e251e1824df None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:55.110963 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 282/559] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:55.117046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dc1a26-a948-4bd3-b4e3-8e251e1824df None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:55.149546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2468123e-a688-4a26-8368-e0e964bc906c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:55.159475 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:55.189278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3046511-7530-47e6-b0a3-88cf352aa6cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:55.189896 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3046511-7530-47e6-b0a3-88cf352aa6cd None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:55.190172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3046511-7530-47e6-b0a3-88cf352aa6cd None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:55.190441 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3046511-7530-47e6-b0a3-88cf352aa6cd None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.191104 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3046511-7530-47e6-b0a3-88cf352aa6cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:55.191537 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3046511-7530-47e6-b0a3-88cf352aa6cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:55.193073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3046511-7530-47e6-b0a3-88cf352aa6cd None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:55.211392 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 278/560] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:55.212680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3046511-7530-47e6-b0a3-88cf352aa6cd None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:55.253774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dc1a26-a948-4bd3-b4e3-8e251e1824df None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:55.289112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aa4eda-1a5f-4491-b83a-586cfbf13e3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:55.289738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aa4eda-1a5f-4491-b83a-586cfbf13e3f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:55.290017 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aa4eda-1a5f-4491-b83a-586cfbf13e3f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:55.290290 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aa4eda-1a5f-4491-b83a-586cfbf13e3f None admin] PATH_INFO: `/v3/users/nova` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.298021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74aa4eda-1a5f-4491-b83a-586cfbf13e3f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:55.299466 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74aa4eda-1a5f-4491-b83a-586cfbf13e3f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-74aa4eda-1a5f-4491-b83a-586cfbf13e3f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:01:55.311754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:55.312721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:55.313636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:55.313636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:55.313636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:01:55.313636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:01:55.313636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:01:55.313636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:01:55.313636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:01:55.313636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:01:55.313636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:01:55.313636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:55.313636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 283/561] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Sep 28 12:01:55 2022] GET /identity/v3/users/nova => generated 82 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:01:55.314204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aa4eda-1a5f-4491-b83a-586cfbf13e3f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:55.321070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3046511-7530-47e6-b0a3-88cf352aa6cd None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:55.324769 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:55.356910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea90286-c6dc-4928-a223-ded44267beee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:55.357526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea90286-c6dc-4928-a223-ded44267beee None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:55.357800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea90286-c6dc-4928-a223-ded44267beee None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:55.358070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea90286-c6dc-4928-a223-ded44267beee None admin] PATH_INFO: `/v3/roles/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.367756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea90286-c6dc-4928-a223-ded44267beee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-2ea90286-c6dc-4928-a223-ded44267beee None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:55.372582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:55.373762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:55.374624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:55.374624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:55.374624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:55.374624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:55.374624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:55.374624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:55.374624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:55.374624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:55.374624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:55.374624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:55.374624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:55.374624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:55.374624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 279/562] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:01:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:55.375307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35c64c8-dff8-46a4-b0f8-68bc5d194333 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:55.375307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35c64c8-dff8-46a4-b0f8-68bc5d194333 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:55.375307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35c64c8-dff8-46a4-b0f8-68bc5d194333 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.380484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 280/563] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:55 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:55.380484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d52a9a1-53e7-4509-9357-2c5c7d165023 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:55.380484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d52a9a1-53e7-4509-9357-2c5c7d165023 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:55.380484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d52a9a1-53e7-4509-9357-2c5c7d165023 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.380484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 281/564] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:55.381121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d52a9a1-53e7-4509-9357-2c5c7d165023 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:55.409053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:55.419759 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aa4eda-1a5f-4491-b83a-586cfbf13e3f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:55.452746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76308aff-a49c-48ea-86f2-a5f9d5d82b12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:55.453370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76308aff-a49c-48ea-86f2-a5f9d5d82b12 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:55.453652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76308aff-a49c-48ea-86f2-a5f9d5d82b12 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:55.453930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76308aff-a49c-48ea-86f2-a5f9d5d82b12 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.461815 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76308aff-a49c-48ea-86f2-a5f9d5d82b12 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:55.463450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76308aff-a49c-48ea-86f2-a5f9d5d82b12 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-76308aff-a49c-48ea-86f2-a5f9d5d82b12 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:01:55.475329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:55.476196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:55.477107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:01:55.477107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:01:55.477107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:01:55.477107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:01:55.477107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:01:55.477107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:01:55.477107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:55.477107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:55.477107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 284/565] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:55.477579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76308aff-a49c-48ea-86f2-a5f9d5d82b12 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:55.489419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d52a9a1-53e7-4509-9357-2c5c7d165023 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:55.523593 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7427a80b-ecc0-47d2-8acf-1162ef0d5730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:55.524197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7427a80b-ecc0-47d2-8acf-1162ef0d5730 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:55.524504 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7427a80b-ecc0-47d2-8acf-1162ef0d5730 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:55.524776 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7427a80b-ecc0-47d2-8acf-1162ef0d5730 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.525501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7427a80b-ecc0-47d2-8acf-1162ef0d5730 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:55.525807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7427a80b-ecc0-47d2-8acf-1162ef0d5730 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:55.527225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7427a80b-ecc0-47d2-8acf-1162ef0d5730 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:55.544216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 282/566] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Sep 28 12:01:55 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:55.545353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df881118-5d85-499f-ab44-7e47df680193 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:55.545643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df881118-5d85-499f-ab44-7e47df680193 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:55.545939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df881118-5d85-499f-ab44-7e47df680193 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.549851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:55.563530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-df881118-5d85-499f-ab44-7e47df680193 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:55.564742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-df881118-5d85-499f-ab44-7e47df680193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:55.583715 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76308aff-a49c-48ea-86f2-a5f9d5d82b12 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:55.617101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3065c99c-c16a-4524-a06a-b5e11942a3d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:55.617693 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3065c99c-c16a-4524-a06a-b5e11942a3d3 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:55.617973 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3065c99c-c16a-4524-a06a-b5e11942a3d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:55.618247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3065c99c-c16a-4524-a06a-b5e11942a3d3 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.618910 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3065c99c-c16a-4524-a06a-b5e11942a3d3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:55.619210 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3065c99c-c16a-4524-a06a-b5e11942a3d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:55.620599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3065c99c-c16a-4524-a06a-b5e11942a3d3 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:55.628333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 285/567] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:01:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:55.629545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bb1246-5aae-4eb0-972f-d1fad15d1c92 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:55.629885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bb1246-5aae-4eb0-972f-d1fad15d1c92 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:55.630111 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bb1246-5aae-4eb0-972f-d1fad15d1c92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.639093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 283/568] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:55.639093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df881118-5d85-499f-ab44-7e47df680193 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:55.650742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:55.661773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-e0bb1246-5aae-4eb0-972f-d1fad15d1c92 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:55.662975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e0bb1246-5aae-4eb0-972f-d1fad15d1c92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:55.734771 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df881118-5d85-499f-ab44-7e47df680193 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:55.737986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 286/569] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:55.741272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bb1246-5aae-4eb0-972f-d1fad15d1c92 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:55.769468 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f57cb39-7676-4955-ac31-7607ed5db4b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:55.770663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f57cb39-7676-4955-ac31-7607ed5db4b7 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:55.770868 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f57cb39-7676-4955-ac31-7607ed5db4b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:55.771112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f57cb39-7676-4955-ac31-7607ed5db4b7 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.771799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f57cb39-7676-4955-ac31-7607ed5db4b7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:55.772148 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f57cb39-7676-4955-ac31-7607ed5db4b7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:55.773562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f57cb39-7676-4955-ac31-7607ed5db4b7 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:55.790308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 284/570] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:01:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:55.790308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f57cb39-7676-4955-ac31-7607ed5db4b7 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:55.819191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:55.872379 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bb1246-5aae-4eb0-972f-d1fad15d1c92 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:55.887483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:55.914609 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007c47f8-d045-4595-bff0-56a934b3258e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:55.915239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007c47f8-d045-4595-bff0-56a934b3258e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:55.915410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f57cb39-7676-4955-ac31-7607ed5db4b7 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:55.915510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007c47f8-d045-4595-bff0-56a934b3258e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:55.915783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007c47f8-d045-4595-bff0-56a934b3258e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.925740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007c47f8-d045-4595-bff0-56a934b3258e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:55.927258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007c47f8-d045-4595-bff0-56a934b3258e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-007c47f8-d045-4595-bff0-56a934b3258e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:55.941377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:55.942388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:55.943274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:55.943274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:55.943274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:55.943274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 287/571] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:55.943274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007c47f8-d045-4595-bff0-56a934b3258e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:55.954104 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5fd89e-287b-4586-80e4-3b662c461160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:55.954731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5fd89e-287b-4586-80e4-3b662c461160 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:55.955016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5fd89e-287b-4586-80e4-3b662c461160 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:55.955292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5fd89e-287b-4586-80e4-3b662c461160 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:55.967094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5fd89e-287b-4586-80e4-3b662c461160 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:55.968727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5fd89e-287b-4586-80e4-3b662c461160 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-dd5fd89e-287b-4586-80e4-3b662c461160 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:55.984629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:55.985626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:55.986500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:55.986500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:55.986500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:55.990865 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 285/572] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:55.991481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5fd89e-287b-4586-80e4-3b662c461160 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:56.047141 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007c47f8-d045-4595-bff0-56a934b3258e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:56.059915 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:56.080571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b751cd-0228-4bf5-a6e6-b06d5f1d3bcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:56.081161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b751cd-0228-4bf5-a6e6-b06d5f1d3bcc None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:56.081500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b751cd-0228-4bf5-a6e6-b06d5f1d3bcc None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:56.081778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b751cd-0228-4bf5-a6e6-b06d5f1d3bcc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:56.082529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b751cd-0228-4bf5-a6e6-b06d5f1d3bcc None admin] RBAC: Adding query filter params (role.id=ed4d262295434a72bb0a8b9a7ff12e71, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=e663aad79bb8429aa223564a15332ee2) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:56.082832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b751cd-0228-4bf5-a6e6-b06d5f1d3bcc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:56.084293 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b751cd-0228-4bf5-a6e6-b06d5f1d3bcc None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:56.089674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 288/573] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:01:55 2022] GET /identity/v3/role_assignments?user.id=e663aad79bb8429aa223564a15332ee2&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=ed4d262295434a72bb0a8b9a7ff12e71 => generated 259 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:56.090444 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b751cd-0228-4bf5-a6e6-b06d5f1d3bcc None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:56.090701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5fd89e-287b-4586-80e4-3b662c461160 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:56.121559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:56.132445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb42210-11ce-45fc-8237-79c4936eb477 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:56.133188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb42210-11ce-45fc-8237-79c4936eb477 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:56.133471 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb42210-11ce-45fc-8237-79c4936eb477 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:56.133746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb42210-11ce-45fc-8237-79c4936eb477 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:56.146489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb42210-11ce-45fc-8237-79c4936eb477 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-0cb42210-11ce-45fc-8237-79c4936eb477 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:56.152022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:56.153238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:56.154586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:56.154586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:56.154586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:56.154586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:56.154586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:56.154586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:56.154586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:56.154586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:56.154586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:56.154586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:56.154586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:56.154586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:56.154586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 286/574] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:01:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:56.154586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb42210-11ce-45fc-8237-79c4936eb477 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:56.214397 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b751cd-0228-4bf5-a6e6-b06d5f1d3bcc None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:56.252251 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c14b94-6429-465c-b946-9db2683afee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:56.252836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c14b94-6429-465c-b946-9db2683afee9 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:56.253119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c14b94-6429-465c-b946-9db2683afee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:56.253393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c14b94-6429-465c-b946-9db2683afee9 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:56.254075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c14b94-6429-465c-b946-9db2683afee9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:56.254386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c14b94-6429-465c-b946-9db2683afee9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:56.255772 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c14b94-6429-465c-b946-9db2683afee9 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:56.270532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb42210-11ce-45fc-8237-79c4936eb477 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:56.272525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 289/575] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:56.273301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c14b94-6429-465c-b946-9db2683afee9 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:56.309916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535aa79f-dd18-497b-8b21-c4c5a2774f9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:56.310517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535aa79f-dd18-497b-8b21-c4c5a2774f9d None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:56.310793 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535aa79f-dd18-497b-8b21-c4c5a2774f9d None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:56.311059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535aa79f-dd18-497b-8b21-c4c5a2774f9d None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:56.311507 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:56.311707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535aa79f-dd18-497b-8b21-c4c5a2774f9d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:56.312003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535aa79f-dd18-497b-8b21-c4c5a2774f9d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:56.313401 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535aa79f-dd18-497b-8b21-c4c5a2774f9d None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:56.326431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 287/576] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:56.327243 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535aa79f-dd18-497b-8b21-c4c5a2774f9d None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:56.382853 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:56.387382 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c14b94-6429-465c-b946-9db2683afee9 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:56.423441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400af1fc-5112-43cc-975e-799a7700b373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:56.424217 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400af1fc-5112-43cc-975e-799a7700b373 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:56.424526 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400af1fc-5112-43cc-975e-799a7700b373 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:56.424800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400af1fc-5112-43cc-975e-799a7700b373 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:56.435009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535aa79f-dd18-497b-8b21-c4c5a2774f9d None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:56.435420 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400af1fc-5112-43cc-975e-799a7700b373 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-400af1fc-5112-43cc-975e-799a7700b373 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:56.440227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:56.441173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:56.442666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:56.442666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:56.442666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:56.442666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:56.442666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:56.442666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:56.442666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:56.442666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:56.442666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:56.442666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:56.442666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:56.442666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:56.442666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 290/577] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:01:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:56.443355 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400af1fc-5112-43cc-975e-799a7700b373 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:56.475509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d417f292-6396-46b6-afd9-e2d2f8fc67d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:56.476482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d417f292-6396-46b6-afd9-e2d2f8fc67d0 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:56.476913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d417f292-6396-46b6-afd9-e2d2f8fc67d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:56.477192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d417f292-6396-46b6-afd9-e2d2f8fc67d0 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:56.477870 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d417f292-6396-46b6-afd9-e2d2f8fc67d0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:56.478169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d417f292-6396-46b6-afd9-e2d2f8fc67d0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:56.479655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d417f292-6396-46b6-afd9-e2d2f8fc67d0 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:56.492630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 288/578] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:01:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:56.492630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d417f292-6396-46b6-afd9-e2d2f8fc67d0 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:56.553553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:56.560146 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400af1fc-5112-43cc-975e-799a7700b373 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:56.597571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067bfe33-a551-4109-a4c9-37879c87561b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:56.598071 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067bfe33-a551-4109-a4c9-37879c87561b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:56.598352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067bfe33-a551-4109-a4c9-37879c87561b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:56.598629 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067bfe33-a551-4109-a4c9-37879c87561b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:56.600882 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d417f292-6396-46b6-afd9-e2d2f8fc67d0 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:56.605508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067bfe33-a551-4109-a4c9-37879c87561b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:56.606949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067bfe33-a551-4109-a4c9-37879c87561b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:56.612036 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-067bfe33-a551-4109-a4c9-37879c87561b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:01:56.618479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:56.619370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:56.620277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:01:56.620277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:01:56.620277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:01:56.620277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:01:56.620277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:01:56.620277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:01:56.620277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:56.620277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:56.620277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 291/579] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:56.620791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067bfe33-a551-4109-a4c9-37879c87561b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:56.646490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab94f5d0-6991-4141-8fa1-b38b205d8282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:56.647191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab94f5d0-6991-4141-8fa1-b38b205d8282 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:56.647459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab94f5d0-6991-4141-8fa1-b38b205d8282 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:56.647648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab94f5d0-6991-4141-8fa1-b38b205d8282 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:56.655192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab94f5d0-6991-4141-8fa1-b38b205d8282 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:56.656646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab94f5d0-6991-4141-8fa1-b38b205d8282 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-ab94f5d0-6991-4141-8fa1-b38b205d8282 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:01:56.667549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:56.673101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:56.674099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:56.674099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:56.674099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:01:56.674099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:01:56.674099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:01:56.674099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:01:56.674099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:01:56.674099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:01:56.674099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 28 12:01:56.674099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:56.674099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 289/580] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:56 2022] GET /identity/v3/users/cinder => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:56.674099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab94f5d0-6991-4141-8fa1-b38b205d8282 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:56.738584 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067bfe33-a551-4109-a4c9-37879c87561b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:56.773708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b935db-7070-4a0a-b8cf-9b8845e84a28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:56.774546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b935db-7070-4a0a-b8cf-9b8845e84a28 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:56.774546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b935db-7070-4a0a-b8cf-9b8845e84a28 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:56.774837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b935db-7070-4a0a-b8cf-9b8845e84a28 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:56.775506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b935db-7070-4a0a-b8cf-9b8845e84a28 None admin] RBAC: Adding query filter params (name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:56.775814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b935db-7070-4a0a-b8cf-9b8845e84a28 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:56.777151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab94f5d0-6991-4141-8fa1-b38b205d8282 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:56.777256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b935db-7070-4a0a-b8cf-9b8845e84a28 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:56.790150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 292/581] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:01:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:56.790150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b935db-7070-4a0a-b8cf-9b8845e84a28 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:56.812294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:56.818634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97775864-c089-4bd3-a0e4-d3d9760c5933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:56.819309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97775864-c089-4bd3-a0e4-d3d9760c5933 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:56.819591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97775864-c089-4bd3-a0e4-d3d9760c5933 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:56.819867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97775864-c089-4bd3-a0e4-d3d9760c5933 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:56.831242 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97775864-c089-4bd3-a0e4-d3d9760c5933 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:56.833322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97775864-c089-4bd3-a0e4-d3d9760c5933 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-97775864-c089-4bd3-a0e4-d3d9760c5933 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:56.848146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:56.849285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:56.850390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:56.850390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:56.850390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:56.853844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 290/582] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:56.854750 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97775864-c089-4bd3-a0e4-d3d9760c5933 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:56.880686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:56.911407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b935db-7070-4a0a-b8cf-9b8845e84a28 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:56.945367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376cbb2a-5254-41a3-80e9-a724088b52e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:56.945988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376cbb2a-5254-41a3-80e9-a724088b52e4 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:56.946268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376cbb2a-5254-41a3-80e9-a724088b52e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:56.946538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376cbb2a-5254-41a3-80e9-a724088b52e4 None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:56.947252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376cbb2a-5254-41a3-80e9-a724088b52e4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:56.947556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376cbb2a-5254-41a3-80e9-a724088b52e4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:56.949088 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376cbb2a-5254-41a3-80e9-a724088b52e4 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:56.963421 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97775864-c089-4bd3-a0e4-d3d9760c5933 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:56.968497 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 293/583] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:01:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:56.968497 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376cbb2a-5254-41a3-80e9-a724088b52e4 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:57.005082 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a41cc2f-3f94-4110-81ac-f2c183b69be9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:57.005963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a41cc2f-3f94-4110-81ac-f2c183b69be9 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:57.006422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a41cc2f-3f94-4110-81ac-f2c183b69be9 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:57.006859 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a41cc2f-3f94-4110-81ac-f2c183b69be9 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:57.007594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a41cc2f-3f94-4110-81ac-f2c183b69be9 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:57.007909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a41cc2f-3f94-4110-81ac-f2c183b69be9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:57.009342 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a41cc2f-3f94-4110-81ac-f2c183b69be9 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:57.027114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 291/584] 10.222.0.26 () {56 vars in 1137 bytes} [Wed Sep 28 12:01:56 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:57.028055 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a41cc2f-3f94-4110-81ac-f2c183b69be9 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:57.055585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:57.078910 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376cbb2a-5254-41a3-80e9-a724088b52e4 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:57.111851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c7e67c-8b43-4feb-9950-7e4dfa3825f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:57.112497 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c7e67c-8b43-4feb-9950-7e4dfa3825f2 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:57.112774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c7e67c-8b43-4feb-9950-7e4dfa3825f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:57.113045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c7e67c-8b43-4feb-9950-7e4dfa3825f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:57.116610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:57.123999 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c7e67c-8b43-4feb-9950-7e4dfa3825f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:57.125771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c7e67c-8b43-4feb-9950-7e4dfa3825f2 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:57.137872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a41cc2f-3f94-4110-81ac-f2c183b69be9 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-a0c7e67c-8b43-4feb-9950-7e4dfa3825f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:57.139883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:57.140816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:57.141695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:57.141695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:57.141695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:57.141695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 294/585] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:57.141695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c7e67c-8b43-4feb-9950-7e4dfa3825f2 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:57.176072 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1810bc3b-b672-4048-be4b-4467e9ab22fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:57.176702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1810bc3b-b672-4048-be4b-4467e9ab22fb None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:57.176980 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1810bc3b-b672-4048-be4b-4467e9ab22fb None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:57.177253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1810bc3b-b672-4048-be4b-4467e9ab22fb None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:57.177890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1810bc3b-b672-4048-be4b-4467e9ab22fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:57.178189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1810bc3b-b672-4048-be4b-4467e9ab22fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:57.179548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1810bc3b-b672-4048-be4b-4467e9ab22fb None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:57.191004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 292/586] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:57.191847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1810bc3b-b672-4048-be4b-4467e9ab22fb None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:57.257271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c7e67c-8b43-4feb-9950-7e4dfa3825f2 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:57.290877 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:57.298516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0a49ea-e706-483e-a614-80ec8ef95066 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:57.299168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0a49ea-e706-483e-a614-80ec8ef95066 None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:57.299444 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0a49ea-e706-483e-a614-80ec8ef95066 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:57.299713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0a49ea-e706-483e-a614-80ec8ef95066 None admin] PATH_INFO: `/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/8bf969348e7a4158acf8fb789430b0e6/roles/ed4d262295434a72bb0a8b9a7ff12e71` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:57.301168 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1810bc3b-b672-4048-be4b-4467e9ab22fb None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:57.317193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0a49ea-e706-483e-a614-80ec8ef95066 None admin] RBAC: Authorizing `identity:create_grant(project_id=8695d562e3e24c7c9300cf3a23cd7502, user_id=8bf969348e7a4158acf8fb789430b0e6, role_id=ed4d262295434a72bb0a8b9a7ff12e71)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:57.318640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0a49ea-e706-483e-a614-80ec8ef95066 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:57.334417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 295/587] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:01:57 2022] PUT /identity/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/8bf969348e7a4158acf8fb789430b0e6/roles/ed4d262295434a72bb0a8b9a7ff12e71 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:01:57.335505 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0a49ea-e706-483e-a614-80ec8ef95066 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:57.350173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1670252a-25da-4156-8834-aa1eb939b98f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:57.351150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1670252a-25da-4156-8834-aa1eb939b98f None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:57.351607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1670252a-25da-4156-8834-aa1eb939b98f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:57.352055 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1670252a-25da-4156-8834-aa1eb939b98f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:57.364920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:57.370181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1670252a-25da-4156-8834-aa1eb939b98f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:57.372914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1670252a-25da-4156-8834-aa1eb939b98f None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-1670252a-25da-4156-8834-aa1eb939b98f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:57.392010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:57.393305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:57.394340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:57.394340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:57.394340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:57.395705 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 293/588] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:57.396769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1670252a-25da-4156-8834-aa1eb939b98f None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:57.458572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0a49ea-e706-483e-a614-80ec8ef95066 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:57.519722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90187987-6d47-4214-9653-7431e2b75fd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:57.520528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90187987-6d47-4214-9653-7431e2b75fd6 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:57.521145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90187987-6d47-4214-9653-7431e2b75fd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:57.521145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90187987-6d47-4214-9653-7431e2b75fd6 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:57.522026 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90187987-6d47-4214-9653-7431e2b75fd6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:57.522386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90187987-6d47-4214-9653-7431e2b75fd6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:57.523862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90187987-6d47-4214-9653-7431e2b75fd6 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:57.531017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1670252a-25da-4156-8834-aa1eb939b98f None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:57.541793 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 296/589] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:57.542426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90187987-6d47-4214-9653-7431e2b75fd6 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:57.570267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9c5c04-be6a-4e7a-8f8a-a52fe5b0a1e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:57.570929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9c5c04-be6a-4e7a-8f8a-a52fe5b0a1e3 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:57.571159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9c5c04-be6a-4e7a-8f8a-a52fe5b0a1e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:57.571426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9c5c04-be6a-4e7a-8f8a-a52fe5b0a1e3 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:57.580410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9c5c04-be6a-4e7a-8f8a-a52fe5b0a1e3 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:57.581862 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9c5c04-be6a-4e7a-8f8a-a52fe5b0a1e3 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:57.592773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-4c9c5c04-be6a-4e7a-8f8a-a52fe5b0a1e3 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:01:57.594321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:57.595288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:57.596218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:57.596218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:57.596218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:01:57.596218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:01:57.596218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:01:57.596218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:01:57.596218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:01:57.596218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:01:57.596218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 28 12:01:57.596218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:57.596218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 294/590] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:01:57 2022] GET /identity/v3/users/glance => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:57.596829 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9c5c04-be6a-4e7a-8f8a-a52fe5b0a1e3 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:57.654098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:57.667751 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90187987-6d47-4214-9653-7431e2b75fd6 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:57.703082 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792e29a4-2231-4628-a1c4-3bd25e1d0c47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:57.703687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792e29a4-2231-4628-a1c4-3bd25e1d0c47 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:57.703967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792e29a4-2231-4628-a1c4-3bd25e1d0c47 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:57.704237 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792e29a4-2231-4628-a1c4-3bd25e1d0c47 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:57.704928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792e29a4-2231-4628-a1c4-3bd25e1d0c47 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:57.705230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792e29a4-2231-4628-a1c4-3bd25e1d0c47 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:57.706852 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792e29a4-2231-4628-a1c4-3bd25e1d0c47 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:57.721087 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 297/591] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:57.722085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792e29a4-2231-4628-a1c4-3bd25e1d0c47 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:57.725717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9c5c04-be6a-4e7a-8f8a-a52fe5b0a1e3 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:57.772535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268babea-bbdd-45ce-9dea-e4c974c1f06c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:57.773770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268babea-bbdd-45ce-9dea-e4c974c1f06c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:57.774623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268babea-bbdd-45ce-9dea-e4c974c1f06c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:57.775103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268babea-bbdd-45ce-9dea-e4c974c1f06c None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:57.786297 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268babea-bbdd-45ce-9dea-e4c974c1f06c None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:57.788972 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268babea-bbdd-45ce-9dea-e4c974c1f06c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-268babea-bbdd-45ce-9dea-e4c974c1f06c None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:01:57.808331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:57.809290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:57.810220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:57.810220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:57.810220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:01:57.810220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:01:57.810220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:01:57.810220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:01:57.810220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:01:57.810220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:01:57.810220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 28 12:01:57.810220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:57.813347 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 295/592] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:01:57 2022] GET /identity/v3/users/neutron => generated 85 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:57.814373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268babea-bbdd-45ce-9dea-e4c974c1f06c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:57.832275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:57.835424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792e29a4-2231-4628-a1c4-3bd25e1d0c47 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:57.872866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bf505e-ea1a-473f-ba55-802026a66f1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:57.873211 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bf505e-ea1a-473f-ba55-802026a66f1c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:57.873591 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bf505e-ea1a-473f-ba55-802026a66f1c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:57.874018 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bf505e-ea1a-473f-ba55-802026a66f1c None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:57.874853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96bf505e-ea1a-473f-ba55-802026a66f1c None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:57.875319 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96bf505e-ea1a-473f-ba55-802026a66f1c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:57.876883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96bf505e-ea1a-473f-ba55-802026a66f1c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:57.896281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 298/593] 10.222.0.26 () {56 vars in 1137 bytes} [Wed Sep 28 12:01:57 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:57.897462 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bf505e-ea1a-473f-ba55-802026a66f1c None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:57.952954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268babea-bbdd-45ce-9dea-e4c974c1f06c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:57.967378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:57.996543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcde9c7-f580-4fdd-8874-3c6094797ec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:57.997801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcde9c7-f580-4fdd-8874-3c6094797ec7 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:57.998227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcde9c7-f580-4fdd-8874-3c6094797ec7 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:57.998640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcde9c7-f580-4fdd-8874-3c6094797ec7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:58.008937 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcde9c7-f580-4fdd-8874-3c6094797ec7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:58.011468 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcde9c7-f580-4fdd-8874-3c6094797ec7 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:58.013410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bf505e-ea1a-473f-ba55-802026a66f1c None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-3bcde9c7-f580-4fdd-8874-3c6094797ec7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:01:58.025828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:58.026919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:58.027912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:01:58.027912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:01:58.027912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:01:58.027912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:01:58.027912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:01:58.027912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:01:58.027912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:58.027912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:58.029143 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 296/594] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:58.030133 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcde9c7-f580-4fdd-8874-3c6094797ec7 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:58.063655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6741a3-060f-4ac8-9943-c5ef8e455438 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:58.064771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6741a3-060f-4ac8-9943-c5ef8e455438 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:58.065219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6741a3-060f-4ac8-9943-c5ef8e455438 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:58.065640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6741a3-060f-4ac8-9943-c5ef8e455438 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:58.067405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6741a3-060f-4ac8-9943-c5ef8e455438 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:58.067875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6741a3-060f-4ac8-9943-c5ef8e455438 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:58.070312 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6741a3-060f-4ac8-9943-c5ef8e455438 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:58.093829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 299/595] 10.222.0.26 () {56 vars in 1139 bytes} [Wed Sep 28 12:01:57 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:58.094964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6741a3-060f-4ac8-9943-c5ef8e455438 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:58.120197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:58.151146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcde9c7-f580-4fdd-8874-3c6094797ec7 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:58.192686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f3723-1d46-476b-b757-c3e9806cc776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:58.193425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f3723-1d46-476b-b757-c3e9806cc776 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:58.193770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f3723-1d46-476b-b757-c3e9806cc776 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:58.194115 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f3723-1d46-476b-b757-c3e9806cc776 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:58.205433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1f3723-1d46-476b-b757-c3e9806cc776 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:58.206951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1f3723-1d46-476b-b757-c3e9806cc776 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-db1f3723-1d46-476b-b757-c3e9806cc776 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:58.222346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:58.223377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:58.224261 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:58.224261 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:58.224261 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:58.224261 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 297/596] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:58.225207 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f3723-1d46-476b-b757-c3e9806cc776 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:58.241987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6741a3-060f-4ac8-9943-c5ef8e455438 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:58.253509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:58.287386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c43eff-1c1c-42bc-9020-f3698c3b5b3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:58.287994 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c43eff-1c1c-42bc-9020-f3698c3b5b3d None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:58.288272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c43eff-1c1c-42bc-9020-f3698c3b5b3d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:58.288566 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c43eff-1c1c-42bc-9020-f3698c3b5b3d None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:58.289261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c43eff-1c1c-42bc-9020-f3698c3b5b3d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:58.289560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c43eff-1c1c-42bc-9020-f3698c3b5b3d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:58.290994 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c43eff-1c1c-42bc-9020-f3698c3b5b3d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:58.303098 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 300/597] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:01:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:58.303888 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c43eff-1c1c-42bc-9020-f3698c3b5b3d None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:58.342601 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f3723-1d46-476b-b757-c3e9806cc776 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:58.379956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:58.381802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f5b4cd-ca29-439f-a380-ab5e17ea1d40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:58.382741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f5b4cd-ca29-439f-a380-ab5e17ea1d40 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:58.383306 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f5b4cd-ca29-439f-a380-ab5e17ea1d40 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:58.383726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f5b4cd-ca29-439f-a380-ab5e17ea1d40 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:58.397277 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f5b4cd-ca29-439f-a380-ab5e17ea1d40 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:58.399556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f5b4cd-ca29-439f-a380-ab5e17ea1d40 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:58.403258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c43eff-1c1c-42bc-9020-f3698c3b5b3d None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-d2f5b4cd-ca29-439f-a380-ab5e17ea1d40 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:58.417112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:58.418107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:58.418974 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:58.418974 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:58.418974 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:58.418974 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 298/598] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:58.419809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6bcd65-9732-4154-b610-4bf4b026b9ab None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:58.419809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6bcd65-9732-4154-b610-4bf4b026b9ab None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:58.419809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6bcd65-9732-4154-b610-4bf4b026b9ab None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:58.421276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 299/599] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:58 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:01:58.421963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6bcd65-9732-4154-b610-4bf4b026b9ab None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:58.447232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49947565-e6ba-4f2b-bebc-da097e5c634a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:58.447838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49947565-e6ba-4f2b-bebc-da097e5c634a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:58.448121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49947565-e6ba-4f2b-bebc-da097e5c634a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:58.448424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49947565-e6ba-4f2b-bebc-da097e5c634a None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:58.449071 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49947565-e6ba-4f2b-bebc-da097e5c634a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:58.449374 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49947565-e6ba-4f2b-bebc-da097e5c634a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:58.450735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49947565-e6ba-4f2b-bebc-da097e5c634a None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:58.467982 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 301/600] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:58.468870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49947565-e6ba-4f2b-bebc-da097e5c634a None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:58.510106 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:58.536749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6bcd65-9732-4154-b610-4bf4b026b9ab None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:58.577049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3d12e2-2197-4794-a19f-9912ed7fdebd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:58.577704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3d12e2-2197-4794-a19f-9912ed7fdebd None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:58.577986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3d12e2-2197-4794-a19f-9912ed7fdebd None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:58.578260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3d12e2-2197-4794-a19f-9912ed7fdebd None admin] PATH_INFO: `/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/2ba80861b08940d398a61879bb70060a/roles/ed4d262295434a72bb0a8b9a7ff12e71` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:58.578360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49947565-e6ba-4f2b-bebc-da097e5c634a None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:58.595316 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3d12e2-2197-4794-a19f-9912ed7fdebd None admin] RBAC: Authorizing `identity:create_grant(project_id=8695d562e3e24c7c9300cf3a23cd7502, user_id=2ba80861b08940d398a61879bb70060a, role_id=ed4d262295434a72bb0a8b9a7ff12e71)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:58.596802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3d12e2-2197-4794-a19f-9912ed7fdebd None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:58.610245 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 300/601] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:01:58 2022] PUT /identity/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/2ba80861b08940d398a61879bb70060a/roles/ed4d262295434a72bb0a8b9a7ff12e71 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:01:58.611727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837cbbdf-3de2-44cc-8da3-59f04b4a9806 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:58.612014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837cbbdf-3de2-44cc-8da3-59f04b4a9806 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:58.612275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837cbbdf-3de2-44cc-8da3-59f04b4a9806 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:58.630946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325bc40b-7af2-475b-943f-dfc8528c41c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:58.631987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325bc40b-7af2-475b-943f-dfc8528c41c9 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:58.632549 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325bc40b-7af2-475b-943f-dfc8528c41c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:58.633146 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325bc40b-7af2-475b-943f-dfc8528c41c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:58.633845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-837cbbdf-3de2-44cc-8da3-59f04b4a9806 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:58.634357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325bc40b-7af2-475b-943f-dfc8528c41c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:58.634844 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325bc40b-7af2-475b-943f-dfc8528c41c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:58.635258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-837cbbdf-3de2-44cc-8da3-59f04b4a9806 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:58.636817 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325bc40b-7af2-475b-943f-dfc8528c41c9 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:58.641401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:58.657927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 302/602] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:58.659966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325bc40b-7af2-475b-943f-dfc8528c41c9 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:58.724208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 301/603] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:01:58.733187 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837cbbdf-3de2-44cc-8da3-59f04b4a9806 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:58.797528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325bc40b-7af2-475b-943f-dfc8528c41c9 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:58.823000 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:58.845418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f2a202-9295-4461-90b6-6763906507e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:58.846359 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f2a202-9295-4461-90b6-6763906507e9 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:58.846927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f2a202-9295-4461-90b6-6763906507e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:58.847383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f2a202-9295-4461-90b6-6763906507e9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:58.858915 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f2a202-9295-4461-90b6-6763906507e9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:58.860728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f2a202-9295-4461-90b6-6763906507e9 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:58.874077 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837cbbdf-3de2-44cc-8da3-59f04b4a9806 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-91f2a202-9295-4461-90b6-6763906507e9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:01:58.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:58.877560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:58.878417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:01:58.878417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:01:58.878417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:01:58.878417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:01:58.878417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:01:58.878417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:01:58.878417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:58.878417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:01:58.881346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 303/604] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:58.888409 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f2a202-9295-4461-90b6-6763906507e9 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:58.924868 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc65ba8d-5cb9-4cae-9ba7-f3cb69f3131c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:58.925417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc65ba8d-5cb9-4cae-9ba7-f3cb69f3131c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:58.925697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc65ba8d-5cb9-4cae-9ba7-f3cb69f3131c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:58.925999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc65ba8d-5cb9-4cae-9ba7-f3cb69f3131c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:58.933159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc65ba8d-5cb9-4cae-9ba7-f3cb69f3131c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:58.934827 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc65ba8d-5cb9-4cae-9ba7-f3cb69f3131c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-fc65ba8d-5cb9-4cae-9ba7-f3cb69f3131c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:01:58.946831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:58.947932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:58.949063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:01:58.949063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:01:58.949063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:01:58.949063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:01:58.949063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:01:58.949063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:01:58.949063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:01:58.949063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:58.949063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 302/605] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:01:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:58.955719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc65ba8d-5cb9-4cae-9ba7-f3cb69f3131c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:58.960174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:59.075591 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f2a202-9295-4461-90b6-6763906507e9 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:59.092807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc65ba8d-5cb9-4cae-9ba7-f3cb69f3131c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:59.110521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:59.126372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f4cbc9-f040-4a89-ac65-6005823b98cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:59.127482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4cbc9-f040-4a89-ac65-6005823b98cf None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:59.127995 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4cbc9-f040-4a89-ac65-6005823b98cf None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:59.128563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4cbc9-f040-4a89-ac65-6005823b98cf None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:59.130270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f4cbc9-f040-4a89-ac65-6005823b98cf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:59.130778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f4cbc9-f040-4a89-ac65-6005823b98cf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:59.130883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650e2a4a-b796-45ea-b4a9-6063610e3091 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:59.131587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650e2a4a-b796-45ea-b4a9-6063610e3091 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:59.131931 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650e2a4a-b796-45ea-b4a9-6063610e3091 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:59.132274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650e2a4a-b796-45ea-b4a9-6063610e3091 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:59.133101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f4cbc9-f040-4a89-ac65-6005823b98cf None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:59.133206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650e2a4a-b796-45ea-b4a9-6063610e3091 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:59.133479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650e2a4a-b796-45ea-b4a9-6063610e3091 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:59.135015 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650e2a4a-b796-45ea-b4a9-6063610e3091 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:59.152503 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 303/607] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:01:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:59.152503 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650e2a4a-b796-45ea-b4a9-6063610e3091 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:59.152789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 304/607] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:01:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:01:59.158719 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f4cbc9-f040-4a89-ac65-6005823b98cf None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:59.266805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650e2a4a-b796-45ea-b4a9-6063610e3091 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:59.280143 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f4cbc9-f040-4a89-ac65-6005823b98cf None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:59.299542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:59.305901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07918c1d-85ca-47d9-a130-aac39bf53ca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:59.306530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07918c1d-85ca-47d9-a130-aac39bf53ca8 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:59.306807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07918c1d-85ca-47d9-a130-aac39bf53ca8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:59.307076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07918c1d-85ca-47d9-a130-aac39bf53ca8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:59.318669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07918c1d-85ca-47d9-a130-aac39bf53ca8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:01:59.320433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604e2614-f59a-4d36-a276-099a8cf1d0ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:59.321298 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604e2614-f59a-4d36-a276-099a8cf1d0ec None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:59.321781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604e2614-f59a-4d36-a276-099a8cf1d0ec None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:59.322276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604e2614-f59a-4d36-a276-099a8cf1d0ec None admin] PATH_INFO: `/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/0e429f061bd04c28b0c60c593b64fb8e/roles/ed4d262295434a72bb0a8b9a7ff12e71` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-07918c1d-85ca-47d9-a130-aac39bf53ca8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:01:59.323539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:59.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:59.325389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:59.325389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:59.325389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:59.325389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:59.325389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:01:59.325389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:01:59.325389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:01:59.325389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:01:59.325389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:01:59.325389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:01:59.325389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:01:59.325389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:59.325389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 304/608] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:01:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:59.326066 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07918c1d-85ca-47d9-a130-aac39bf53ca8 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:59.342542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604e2614-f59a-4d36-a276-099a8cf1d0ec None admin] RBAC: Authorizing `identity:create_grant(project_id=8695d562e3e24c7c9300cf3a23cd7502, user_id=0e429f061bd04c28b0c60c593b64fb8e, role_id=ed4d262295434a72bb0a8b9a7ff12e71)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:59.344398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604e2614-f59a-4d36-a276-099a8cf1d0ec None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:59.364852 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 305/609] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:01:59 2022] PUT /identity/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/0e429f061bd04c28b0c60c593b64fb8e/roles/ed4d262295434a72bb0a8b9a7ff12e71 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:01:59.366422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:59.367951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604e2614-f59a-4d36-a276-099a8cf1d0ec None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:59.456344 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07918c1d-85ca-47d9-a130-aac39bf53ca8 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:59.503951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a75c3fb-eb21-4312-9b6b-5212b624aa1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:59.505984 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a75c3fb-eb21-4312-9b6b-5212b624aa1f None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:59.506380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a75c3fb-eb21-4312-9b6b-5212b624aa1f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:59.506755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a75c3fb-eb21-4312-9b6b-5212b624aa1f None admin] PATH_INFO: `/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/815504693bcc40b6aef06aacb2a9a35b/roles/ed4d262295434a72bb0a8b9a7ff12e71` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:59.527782 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a75c3fb-eb21-4312-9b6b-5212b624aa1f None admin] RBAC: Authorizing `identity:create_grant(project_id=8695d562e3e24c7c9300cf3a23cd7502, user_id=815504693bcc40b6aef06aacb2a9a35b, role_id=ed4d262295434a72bb0a8b9a7ff12e71)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:59.529389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a75c3fb-eb21-4312-9b6b-5212b624aa1f None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:59.541329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604e2614-f59a-4d36-a276-099a8cf1d0ec None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:59.556617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 305/610] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:01:59 2022] PUT /identity/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/815504693bcc40b6aef06aacb2a9a35b/roles/ed4d262295434a72bb0a8b9a7ff12e71 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:01:59.598868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b821797-f6ed-4853-943c-18b494541686 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:59.599506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b821797-f6ed-4853-943c-18b494541686 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:59.599780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b821797-f6ed-4853-943c-18b494541686 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:59.600051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b821797-f6ed-4853-943c-18b494541686 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:59.600798 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b821797-f6ed-4853-943c-18b494541686 None admin] RBAC: Adding query filter params (name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:01:59.601106 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b821797-f6ed-4853-943c-18b494541686 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:59.602578 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b821797-f6ed-4853-943c-18b494541686 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:59.615589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 306/611] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:01:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:01:59.621458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a75c3fb-eb21-4312-9b6b-5212b624aa1f None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:59.752516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9b4ad9-0762-4ed8-8844-b4878c04e362 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:59.752844 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9b4ad9-0762-4ed8-8844-b4878c04e362 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:59.753071 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9b4ad9-0762-4ed8-8844-b4878c04e362 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:59.758970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 307/612] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:01:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:01:59.760560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f82e5d4-289a-4521-a607-2c62a5f1f0d2 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:59.760839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f82e5d4-289a-4521-a607-2c62a5f1f0d2 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:59.761113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f82e5d4-289a-4521-a607-2c62a5f1f0d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:59.762749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:59.786322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a75c3fb-eb21-4312-9b6b-5212b624aa1f None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:01:59.793442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-2f82e5d4-289a-4521-a607-2c62a5f1f0d2 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:01:59.794701 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2f82e5d4-289a-4521-a607-2c62a5f1f0d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:01:59.822797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:01:59.830774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49467956-c930-477f-9e02-e6caa238ad52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:01:59.832902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49467956-c930-477f-9e02-e6caa238ad52 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:01:59.833728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49467956-c930-477f-9e02-e6caa238ad52 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:01:59.834433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49467956-c930-477f-9e02-e6caa238ad52 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:01:59.846515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49467956-c930-477f-9e02-e6caa238ad52 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:01:59.848415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49467956-c930-477f-9e02-e6caa238ad52 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-49467956-c930-477f-9e02-e6caa238ad52 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:01:59.867564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:01:59.868552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:01:59.869637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:01:59.869637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:01:59.869637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:01:59.872191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 306/613] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:01:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:01:59.875577 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 308/614] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:01:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:01:59.881359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49467956-c930-477f-9e02-e6caa238ad52 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:01:59.998445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49467956-c930-477f-9e02-e6caa238ad52 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:00.046856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5835497b-53b7-4057-905b-1318a0fd6f58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:00.047887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5835497b-53b7-4057-905b-1318a0fd6f58 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:00.048367 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5835497b-53b7-4057-905b-1318a0fd6f58 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:00.048840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5835497b-53b7-4057-905b-1318a0fd6f58 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:00.049665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5835497b-53b7-4057-905b-1318a0fd6f58 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:00.050149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5835497b-53b7-4057-905b-1318a0fd6f58 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:00.051767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5835497b-53b7-4057-905b-1318a0fd6f58 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:00.062270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:00.069646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 307/615] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:01:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:00.074503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f82e5d4-289a-4521-a607-2c62a5f1f0d2 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:00.186746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f82e5d4-289a-4521-a607-2c62a5f1f0d2 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:00.224183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5835497b-53b7-4057-905b-1318a0fd6f58 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:00.231731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f59fd6-c396-4f12-97c3-b7088315b527 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:00.232661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f59fd6-c396-4f12-97c3-b7088315b527 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:00.233239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f59fd6-c396-4f12-97c3-b7088315b527 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:00.233688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f59fd6-c396-4f12-97c3-b7088315b527 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:00.245447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f59fd6-c396-4f12-97c3-b7088315b527 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:00.247175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f59fd6-c396-4f12-97c3-b7088315b527 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-b2f59fd6-c396-4f12-97c3-b7088315b527 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:00.260538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:00.261818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:00.263383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:00.263383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:00.263383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:00.263383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:00.263383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:00.263383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:00.263383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:00.263383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:00.263383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 28 12:02:00.263383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:00.266378 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 309/616] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:02:00 2022] GET /identity/v3/users/swift => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:00.276130 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f59fd6-c396-4f12-97c3-b7088315b527 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:00.320753 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:00.384825 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5835497b-53b7-4057-905b-1318a0fd6f58 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:00.460265 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208b65e2-ee64-4b4b-a34e-b114fb48f6c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:00.461316 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208b65e2-ee64-4b4b-a34e-b114fb48f6c8 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:00.462016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208b65e2-ee64-4b4b-a34e-b114fb48f6c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:00.463506 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208b65e2-ee64-4b4b-a34e-b114fb48f6c8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:00.487980 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208b65e2-ee64-4b4b-a34e-b114fb48f6c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-208b65e2-ee64-4b4b-a34e-b114fb48f6c8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:00.505218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:00.506304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:00.507171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:00.507171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:00.507171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:00.507171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:00.507171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:00.507171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:00.507171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:00.507171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:00.507171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:00.507171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:00.507171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:00.507171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:00.507828 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f59fd6-c396-4f12-97c3-b7088315b527 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:00.507910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 308/617] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:02:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:00.508923 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208b65e2-ee64-4b4b-a34e-b114fb48f6c8 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:00.577594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:00.609645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df52b23-995c-4bcc-afc9-c3fac6196341 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:00.610256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df52b23-995c-4bcc-afc9-c3fac6196341 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:00.610535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df52b23-995c-4bcc-afc9-c3fac6196341 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:00.610806 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df52b23-995c-4bcc-afc9-c3fac6196341 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:00.611544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df52b23-995c-4bcc-afc9-c3fac6196341 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:00.611848 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df52b23-995c-4bcc-afc9-c3fac6196341 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:00.613382 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df52b23-995c-4bcc-afc9-c3fac6196341 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:00.648524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208b65e2-ee64-4b4b-a34e-b114fb48f6c8 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:00.662820 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 310/618] 10.222.0.26 () {56 vars in 1135 bytes} [Wed Sep 28 12:02:00 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 387 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:00.668340 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df52b23-995c-4bcc-afc9-c3fac6196341 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:00.698417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8040fee-7d91-4dbf-9924-9227ed4e9d03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:00.699056 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8040fee-7d91-4dbf-9924-9227ed4e9d03 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:00.699340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8040fee-7d91-4dbf-9924-9227ed4e9d03 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:00.699621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8040fee-7d91-4dbf-9924-9227ed4e9d03 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:00.700379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8040fee-7d91-4dbf-9924-9227ed4e9d03 None admin] RBAC: Adding query filter params (name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:00.700713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8040fee-7d91-4dbf-9924-9227ed4e9d03 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:00.702166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8040fee-7d91-4dbf-9924-9227ed4e9d03 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:00.713773 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 309/619] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:02:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:00.720561 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8040fee-7d91-4dbf-9924-9227ed4e9d03 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:00.844738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:00.852643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8040fee-7d91-4dbf-9924-9227ed4e9d03 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:00.874763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:00.898400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df52b23-995c-4bcc-afc9-c3fac6196341 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:00.900982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfe1931-bd59-4876-9b97-0f635af26d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:00.901610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfe1931-bd59-4876-9b97-0f635af26d94 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:00.901888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfe1931-bd59-4876-9b97-0f635af26d94 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:00.902162 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfe1931-bd59-4876-9b97-0f635af26d94 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:00.927826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfe1931-bd59-4876-9b97-0f635af26d94 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:00.929318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfe1931-bd59-4876-9b97-0f635af26d94 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-5bfe1931-bd59-4876-9b97-0f635af26d94 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:00.959192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:00.960201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:00.961128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:00.961128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:00.961128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:00.962310 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 310/620] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:00.969959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff48d23-cc6f-4cd4-979c-01f09eb390f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:00.970290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfe1931-bd59-4876-9b97-0f635af26d94 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:00.970595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff48d23-cc6f-4cd4-979c-01f09eb390f2 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:00.970876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff48d23-cc6f-4cd4-979c-01f09eb390f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:00.971151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff48d23-cc6f-4cd4-979c-01f09eb390f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:00.994243 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff48d23-cc6f-4cd4-979c-01f09eb390f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:00.995701 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff48d23-cc6f-4cd4-979c-01f09eb390f2 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-dff48d23-cc6f-4cd4-979c-01f09eb390f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:01.023212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:01.024185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:01.025086 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:01.025086 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:01.025086 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:01.028807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 311/621] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 361 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:01.033786 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff48d23-cc6f-4cd4-979c-01f09eb390f2 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:01.078317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfe1931-bd59-4876-9b97-0f635af26d94 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:01.125263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c4a03e-edf8-4b08-a508-2df5c0087022 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:01.126666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c4a03e-edf8-4b08-a508-2df5c0087022 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:01.126954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c4a03e-edf8-4b08-a508-2df5c0087022 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:01.127235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c4a03e-edf8-4b08-a508-2df5c0087022 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:01.127916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c4a03e-edf8-4b08-a508-2df5c0087022 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:01.128222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c4a03e-edf8-4b08-a508-2df5c0087022 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:01.129766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c4a03e-edf8-4b08-a508-2df5c0087022 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:01.170590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 311/622] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:01.181367 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c4a03e-edf8-4b08-a508-2df5c0087022 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:01.219168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff48d23-cc6f-4cd4-979c-01f09eb390f2 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:01.228932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:01.248766 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:01.260932 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca4771d-4693-4991-b061-72388712dca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:01.262519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca4771d-4693-4991-b061-72388712dca8 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:01.263012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca4771d-4693-4991-b061-72388712dca8 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:01.263620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca4771d-4693-4991-b061-72388712dca8 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:01.264286 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca4771d-4693-4991-b061-72388712dca8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:01.264631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca4771d-4693-4991-b061-72388712dca8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:01.266296 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca4771d-4693-4991-b061-72388712dca8 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:01.283176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 312/623] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:01.285689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250add03-48c0-4010-8314-9810665b00a2 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:01.285978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250add03-48c0-4010-8314-9810665b00a2 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:01.286251 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250add03-48c0-4010-8314-9810665b00a2 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:01.287732 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 313/624] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:01.289902 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250add03-48c0-4010-8314-9810665b00a2 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:01.315714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c4a03e-edf8-4b08-a508-2df5c0087022 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:01.357431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5eab52-9ccc-41ca-90a0-d826eced4341 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:01.358328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5eab52-9ccc-41ca-90a0-d826eced4341 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:01.358887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5eab52-9ccc-41ca-90a0-d826eced4341 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:01.359388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5eab52-9ccc-41ca-90a0-d826eced4341 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:01.369858 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5eab52-9ccc-41ca-90a0-d826eced4341 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:01.371346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5eab52-9ccc-41ca-90a0-d826eced4341 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-6a5eab52-9ccc-41ca-90a0-d826eced4341 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:01.386851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:01.387807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:01.388784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:01.388784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:01.388784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:01.388784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:01.388784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:01.388784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:01.388784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:01.388784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:01.388784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:02:01.388784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:01.392370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 312/625] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Sep 28 12:02:01 2022] GET /identity/v3/users/nova => generated 82 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:01.395834 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a792f26b-7c0d-4b35-a1ff-431ed42a19ad None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:01.396123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a792f26b-7c0d-4b35-a1ff-431ed42a19ad None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:01.396432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a792f26b-7c0d-4b35-a1ff-431ed42a19ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:01.414621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250add03-48c0-4010-8314-9810665b00a2 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:01.419489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-a792f26b-7c0d-4b35-a1ff-431ed42a19ad None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:01.420697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a792f26b-7c0d-4b35-a1ff-431ed42a19ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:01.451202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace38af0-a82d-4b20-845b-03e0fdd66fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:01.452194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace38af0-a82d-4b20-845b-03e0fdd66fc5 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:01.452723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace38af0-a82d-4b20-845b-03e0fdd66fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:01.453207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace38af0-a82d-4b20-845b-03e0fdd66fc5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:01.461336 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace38af0-a82d-4b20-845b-03e0fdd66fc5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:01.463316 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace38af0-a82d-4b20-845b-03e0fdd66fc5 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-ace38af0-a82d-4b20-845b-03e0fdd66fc5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:01.477139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:01.478065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:01.478919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:01.478919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:01.478919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:01.478919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:01.478919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:01.478919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:01.478919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:01.478919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:01.483735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 314/626] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:01.483878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace38af0-a82d-4b20-845b-03e0fdd66fc5 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:01.523860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 313/627] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:02:01.525165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a792f26b-7c0d-4b35-a1ff-431ed42a19ad None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:01.541341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:01.561360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:01.601738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace38af0-a82d-4b20-845b-03e0fdd66fc5 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:01.640625 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc0c860-873f-487b-a065-903d296186cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:01.642801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc0c860-873f-487b-a065-903d296186cc None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:01.643471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc0c860-873f-487b-a065-903d296186cc None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:01.645130 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc0c860-873f-487b-a065-903d296186cc None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:01.646275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc0c860-873f-487b-a065-903d296186cc None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:01.650860 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc0c860-873f-487b-a065-903d296186cc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:01.652601 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc0c860-873f-487b-a065-903d296186cc None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:01.680695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 315/628] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Sep 28 12:02:01 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:01.686386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc0c860-873f-487b-a065-903d296186cc None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:01.718527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a792f26b-7c0d-4b35-a1ff-431ed42a19ad None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:01.794824 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc0c860-873f-487b-a065-903d296186cc None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:01.803467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:01.817585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3688d1-e2ac-4240-9d34-23b946ee5b48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:01.818214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3688d1-e2ac-4240-9d34-23b946ee5b48 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:01.818498 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3688d1-e2ac-4240-9d34-23b946ee5b48 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:01.818778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3688d1-e2ac-4240-9d34-23b946ee5b48 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:01.819484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3688d1-e2ac-4240-9d34-23b946ee5b48 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:01.819791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3688d1-e2ac-4240-9d34-23b946ee5b48 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:01.821306 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3688d1-e2ac-4240-9d34-23b946ee5b48 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:01.834514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d52617-ec8d-4961-b793-6e998097561d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:01.835416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d52617-ec8d-4961-b793-6e998097561d None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:01.835913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d52617-ec8d-4961-b793-6e998097561d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:01.836433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d52617-ec8d-4961-b793-6e998097561d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:01.850807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 314/629] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:01.856535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3688d1-e2ac-4240-9d34-23b946ee5b48 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:01.908040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d52617-ec8d-4961-b793-6e998097561d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:01.910082 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d52617-ec8d-4961-b793-6e998097561d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:01.982259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:02.011980 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3688d1-e2ac-4240-9d34-23b946ee5b48 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:02.036341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 316/630] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:02:02.036341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d52617-ec8d-4961-b793-6e998097561d None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:02.042967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:02.056524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fef1104-5373-47f7-8321-977759ee4270 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:02.057508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fef1104-5373-47f7-8321-977759ee4270 None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:02.057999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fef1104-5373-47f7-8321-977759ee4270 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:02.058526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fef1104-5373-47f7-8321-977759ee4270 None admin] PATH_INFO: `/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/e663aad79bb8429aa223564a15332ee2/roles/ed4d262295434a72bb0a8b9a7ff12e71` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:02.081008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fef1104-5373-47f7-8321-977759ee4270 None admin] RBAC: Authorizing `identity:create_grant(project_id=8695d562e3e24c7c9300cf3a23cd7502, user_id=e663aad79bb8429aa223564a15332ee2, role_id=ed4d262295434a72bb0a8b9a7ff12e71)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:02.082612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fef1104-5373-47f7-8321-977759ee4270 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:02.105299 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 315/631] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:02:01 2022] PUT /identity/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/e663aad79bb8429aa223564a15332ee2/roles/ed4d262295434a72bb0a8b9a7ff12e71 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:02:02.106355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fef1104-5373-47f7-8321-977759ee4270 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:02.166304 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d52617-ec8d-4961-b793-6e998097561d None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:02.209343 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c7ea27-fbe6-41ce-9694-6d4d2de2aa7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:02.210482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c7ea27-fbe6-41ce-9694-6d4d2de2aa7b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:02.210977 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c7ea27-fbe6-41ce-9694-6d4d2de2aa7b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:02.211434 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c7ea27-fbe6-41ce-9694-6d4d2de2aa7b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:02.224734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c7ea27-fbe6-41ce-9694-6d4d2de2aa7b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:02.232636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fef1104-5373-47f7-8321-977759ee4270 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-e0c7ea27-fbe6-41ce-9694-6d4d2de2aa7b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:02.232841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:02.233801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:02.234631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:02.234631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:02.234631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:02.234631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:02.234631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:02.234631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:02.234631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:02.234631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:02.234631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:02.234631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:02.234631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:02.234631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:02.234631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 317/632] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:02:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:02.235938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701df8d5-6a44-4ebf-bddb-c7cadefa032b None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:02.236060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701df8d5-6a44-4ebf-bddb-c7cadefa032b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:02.236254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701df8d5-6a44-4ebf-bddb-c7cadefa032b None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:02.243599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 318/633] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:02.243599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca71bd6-46be-47d5-930c-9a078e4a83ba None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:02.243599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca71bd6-46be-47d5-930c-9a078e4a83ba None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:02.243599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca71bd6-46be-47d5-930c-9a078e4a83ba None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:02.243599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 319/634] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:02.243599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca71bd6-46be-47d5-930c-9a078e4a83ba None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:02.274976 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:02.281529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1564dfaa-ef01-484f-be37-de09a207d669 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:02.282518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1564dfaa-ef01-484f-be37-de09a207d669 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:02.282813 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1564dfaa-ef01-484f-be37-de09a207d669 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:02.283084 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1564dfaa-ef01-484f-be37-de09a207d669 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:02.298307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1564dfaa-ef01-484f-be37-de09a207d669 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:02.299765 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1564dfaa-ef01-484f-be37-de09a207d669 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-1564dfaa-ef01-484f-be37-de09a207d669 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:02.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:02.320125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:02.321121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:02.321121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:02.321121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:02.321121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 316/635] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:02.321657 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f4cf2c-1b4c-4342-9517-4e2eafd051c4 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:02.322389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f4cf2c-1b4c-4342-9517-4e2eafd051c4 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:02.322743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f4cf2c-1b4c-4342-9517-4e2eafd051c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:02.333183 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:02.342226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-12f4cf2c-1b4c-4342-9517-4e2eafd051c4 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:02.343409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-12f4cf2c-1b4c-4342-9517-4e2eafd051c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:02.378368 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca71bd6-46be-47d5-930c-9a078e4a83ba None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:02.417120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 317/636] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:02:02.417305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1709ea6e-baa9-4a93-a662-126556c106be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:02.417798 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1709ea6e-baa9-4a93-a662-126556c106be None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:02.418142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1709ea6e-baa9-4a93-a662-126556c106be None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:02.418503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1709ea6e-baa9-4a93-a662-126556c106be None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:02.419239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1709ea6e-baa9-4a93-a662-126556c106be None admin] RBAC: Adding query filter params (name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:02.419599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1709ea6e-baa9-4a93-a662-126556c106be None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:02.419878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81860027-4659-43aa-8db5-baf833c97e0e None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:02.420354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81860027-4659-43aa-8db5-baf833c97e0e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:02.420804 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81860027-4659-43aa-8db5-baf833c97e0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:02.421334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1709ea6e-baa9-4a93-a662-126556c106be None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:02.437944 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 320/637] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:02:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:02.437944 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1709ea6e-baa9-4a93-a662-126556c106be None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:02.445684 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-81860027-4659-43aa-8db5-baf833c97e0e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:02.447111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-81860027-4659-43aa-8db5-baf833c97e0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:02.530153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 318/638] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:02:02.530153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81860027-4659-43aa-8db5-baf833c97e0e None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:02.574008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:02.591802 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1709ea6e-baa9-4a93-a662-126556c106be None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:02.632517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81860027-4659-43aa-8db5-baf833c97e0e None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:02.669386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:02.670344 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c142f9c-095c-4930-986a-cee098539eeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:02.671415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c142f9c-095c-4930-986a-cee098539eeb None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:02.671951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c142f9c-095c-4930-986a-cee098539eeb None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:02.672875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c142f9c-095c-4930-986a-cee098539eeb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:02.684067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c142f9c-095c-4930-986a-cee098539eeb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:02.685772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c142f9c-095c-4930-986a-cee098539eeb None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:02.700995 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d4c038-4d76-4d11-9bcf-34e20829f591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-0c142f9c-095c-4930-986a-cee098539eeb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:02.701339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:02.702326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:02.703196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:02.703196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:02.703196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:02.703362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d4c038-4d76-4d11-9bcf-34e20829f591 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:02.703362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d4c038-4d76-4d11-9bcf-34e20829f591 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:02.703362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d4c038-4d76-4d11-9bcf-34e20829f591 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:02.703362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d4c038-4d76-4d11-9bcf-34e20829f591 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:02.703362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d4c038-4d76-4d11-9bcf-34e20829f591 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:02.704524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d4c038-4d76-4d11-9bcf-34e20829f591 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:02.705808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 319/639] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:02.717316 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c142f9c-095c-4930-986a-cee098539eeb None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:02.726551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 321/640] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:02.727726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d4c038-4d76-4d11-9bcf-34e20829f591 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:02.840527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:02.843261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d4c038-4d76-4d11-9bcf-34e20829f591 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:02.850271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c142f9c-095c-4930-986a-cee098539eeb None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:02.885224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae50c8fa-d6d5-472c-81cd-a19c787fd1cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:02.885832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae50c8fa-d6d5-472c-81cd-a19c787fd1cb None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:02.886116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae50c8fa-d6d5-472c-81cd-a19c787fd1cb None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:02.886392 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae50c8fa-d6d5-472c-81cd-a19c787fd1cb None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:02.887042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae50c8fa-d6d5-472c-81cd-a19c787fd1cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:02.887349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae50c8fa-d6d5-472c-81cd-a19c787fd1cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:02.888710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fbfc73-da73-4ccc-988b-e215a2c5b63e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:02.889154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae50c8fa-d6d5-472c-81cd-a19c787fd1cb None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:02.889340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fbfc73-da73-4ccc-988b-e215a2c5b63e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:02.889618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fbfc73-da73-4ccc-988b-e215a2c5b63e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:02.889882 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fbfc73-da73-4ccc-988b-e215a2c5b63e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:02.900749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46fbfc73-da73-4ccc-988b-e215a2c5b63e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:02.903738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 322/641] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:02.904895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae50c8fa-d6d5-472c-81cd-a19c787fd1cb None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-46fbfc73-da73-4ccc-988b-e215a2c5b63e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:02.905915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:02.907038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:02.908068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:02.908068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:02.908068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:02.908068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:02.908068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:02.908068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:02.908068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:02.908068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:02.908068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:02.908068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:02.908068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:02.908068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:02.911764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 320/642] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:02:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:02.912634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fbfc73-da73-4ccc-988b-e215a2c5b63e None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:02.960779 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:03.021340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fbfc73-da73-4ccc-988b-e215a2c5b63e None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:03.025618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae50c8fa-d6d5-472c-81cd-a19c787fd1cb None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:03.059469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3391a9c-9205-4106-9b43-e36b95cdae58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:03.060401 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3391a9c-9205-4106-9b43-e36b95cdae58 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:03.060754 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3391a9c-9205-4106-9b43-e36b95cdae58 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:03.061340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3391a9c-9205-4106-9b43-e36b95cdae58 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:03.065699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209e4e7b-3635-4dc9-97d1-4820b4175f6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:03.066410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209e4e7b-3635-4dc9-97d1-4820b4175f6d None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:03.066631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209e4e7b-3635-4dc9-97d1-4820b4175f6d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:03.066904 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209e4e7b-3635-4dc9-97d1-4820b4175f6d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:03.071803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3391a9c-9205-4106-9b43-e36b95cdae58 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:03.074137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209e4e7b-3635-4dc9-97d1-4820b4175f6d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:03.075601 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209e4e7b-3635-4dc9-97d1-4820b4175f6d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-f3391a9c-9205-4106-9b43-e36b95cdae58 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:03.077059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:03.078046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:03.078908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:03.078908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:03.078908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:03.078908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:03.078908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:03.078908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:03.078908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:03.078908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:03.078908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:03.078908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:03.078908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:03.078908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:03.078908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 321/643] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:02:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:03.079599 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3391a9c-9205-4106-9b43-e36b95cdae58 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-209e4e7b-3635-4dc9-97d1-4820b4175f6d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:03.093459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:03.094370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:03.095439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:03.095439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:03.095439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:03.095439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:03.095439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:03.095439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:03.095439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:03.095439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:03.095439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 323/644] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:03.095439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209e4e7b-3635-4dc9-97d1-4820b4175f6d None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:03.103386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:03.190825 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3391a9c-9205-4106-9b43-e36b95cdae58 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:03.200669 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209e4e7b-3635-4dc9-97d1-4820b4175f6d None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:03.210172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:03.227198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a457ded-939b-4294-9ea6-99d53a975084 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:03.227785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a457ded-939b-4294-9ea6-99d53a975084 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:03.228037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a457ded-939b-4294-9ea6-99d53a975084 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:03.228321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a457ded-939b-4294-9ea6-99d53a975084 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:03.235610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a457ded-939b-4294-9ea6-99d53a975084 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:03.237099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a457ded-939b-4294-9ea6-99d53a975084 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:03.239239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2dd48f3-dfb4-4e91-9239-361cef4afe77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:03.239841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dd48f3-dfb4-4e91-9239-361cef4afe77 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:03.240119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dd48f3-dfb4-4e91-9239-361cef4afe77 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:03.240428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dd48f3-dfb4-4e91-9239-361cef4afe77 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:03.241503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2dd48f3-dfb4-4e91-9239-361cef4afe77 None admin] RBAC: Adding query filter params (name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:03.241503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2dd48f3-dfb4-4e91-9239-361cef4afe77 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:03.242739 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2dd48f3-dfb4-4e91-9239-361cef4afe77 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:03.255165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 324/645] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:02:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-3a457ded-939b-4294-9ea6-99d53a975084 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:03.255319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:03.256202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:03.257188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:03.257188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:03.257188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:03.257188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:03.257188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:03.257188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:03.257188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:03.257188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:03.257188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 28 12:02:03.257188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:03.257724 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2dd48f3-dfb4-4e91-9239-361cef4afe77 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:03.257830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 322/646] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:02:03 2022] GET /identity/v3/users/cinder => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:03.257830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a457ded-939b-4294-9ea6-99d53a975084 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:03.345157 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:03.369598 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a457ded-939b-4294-9ea6-99d53a975084 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:03.370685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2dd48f3-dfb4-4e91-9239-361cef4afe77 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:03.404770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4231f959-0275-4360-b249-6cb04ca587f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:03.405377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4231f959-0275-4360-b249-6cb04ca587f6 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:03.405660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4231f959-0275-4360-b249-6cb04ca587f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:03.405930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4231f959-0275-4360-b249-6cb04ca587f6 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:03.406622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4231f959-0275-4360-b249-6cb04ca587f6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:03.406923 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4231f959-0275-4360-b249-6cb04ca587f6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:03.408371 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4231f959-0275-4360-b249-6cb04ca587f6 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:03.413289 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a4e20b-1627-47d0-8458-f26d5a1c98ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:03.413891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a4e20b-1627-47d0-8458-f26d5a1c98ce None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:03.414174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a4e20b-1627-47d0-8458-f26d5a1c98ce None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:03.414450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a4e20b-1627-47d0-8458-f26d5a1c98ce None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:03.415121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a4e20b-1627-47d0-8458-f26d5a1c98ce None admin] RBAC: Adding query filter params (name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:03.415428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a4e20b-1627-47d0-8458-f26d5a1c98ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:03.416983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a4e20b-1627-47d0-8458-f26d5a1c98ce None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:03.426058 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 323/647] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:03.426058 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4231f959-0275-4360-b249-6cb04ca587f6 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:03.426353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 325/648] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:02:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:03.426353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a4e20b-1627-47d0-8458-f26d5a1c98ce None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:03.460689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:03.539987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a4e20b-1627-47d0-8458-f26d5a1c98ce None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:03.544026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4231f959-0275-4360-b249-6cb04ca587f6 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:03.575025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b642b3-df81-4f77-8497-7bb876a7a75a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:03.575748 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b642b3-df81-4f77-8497-7bb876a7a75a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:03.576134 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b642b3-df81-4f77-8497-7bb876a7a75a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:03.576550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b642b3-df81-4f77-8497-7bb876a7a75a None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:03.577401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b642b3-df81-4f77-8497-7bb876a7a75a None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:03.577715 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b642b3-df81-4f77-8497-7bb876a7a75a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:03.579188 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b642b3-df81-4f77-8497-7bb876a7a75a None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:03.579365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcb0a85-cd4b-407f-8772-3427759a6cca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:03.579965 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcb0a85-cd4b-407f-8772-3427759a6cca None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:03.580241 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcb0a85-cd4b-407f-8772-3427759a6cca None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:03.580723 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcb0a85-cd4b-407f-8772-3427759a6cca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:03.585194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:03.594965 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcb0a85-cd4b-407f-8772-3427759a6cca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:03.596418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcb0a85-cd4b-407f-8772-3427759a6cca None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:03.602061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 326/649] 10.222.0.26 () {56 vars in 1137 bytes} [Wed Sep 28 12:02:03 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:03.602061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b642b3-df81-4f77-8497-7bb876a7a75a None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-bbcb0a85-cd4b-407f-8772-3427759a6cca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:03.617962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:03.618917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:03.619787 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:03.619787 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:03.619787 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:03.624080 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 324/650] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:03.624926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcb0a85-cd4b-407f-8772-3427759a6cca None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:03.698615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:03.710339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b642b3-df81-4f77-8497-7bb876a7a75a None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:03.730331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcb0a85-cd4b-407f-8772-3427759a6cca None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:03.746318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638f65e8-7f5a-4cbf-8749-ef9c3fdf5662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:03.746841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638f65e8-7f5a-4cbf-8749-ef9c3fdf5662 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:03.747111 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638f65e8-7f5a-4cbf-8749-ef9c3fdf5662 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:03.747391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638f65e8-7f5a-4cbf-8749-ef9c3fdf5662 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:03.748110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638f65e8-7f5a-4cbf-8749-ef9c3fdf5662 None admin] RBAC: Adding query filter params (role.id=ed4d262295434a72bb0a8b9a7ff12e71, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=8bf969348e7a4158acf8fb789430b0e6) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:03.748449 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638f65e8-7f5a-4cbf-8749-ef9c3fdf5662 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:03.749863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638f65e8-7f5a-4cbf-8749-ef9c3fdf5662 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:03.755676 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 327/651] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:03 2022] GET /identity/v3/role_assignments?user.id=8bf969348e7a4158acf8fb789430b0e6&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=ed4d262295434a72bb0a8b9a7ff12e71 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:03.756497 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638f65e8-7f5a-4cbf-8749-ef9c3fdf5662 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:03.781226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe6ba95-cf6d-44f9-a5e6-2bb6033f9180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:03.782386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe6ba95-cf6d-44f9-a5e6-2bb6033f9180 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:03.782817 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe6ba95-cf6d-44f9-a5e6-2bb6033f9180 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:03.783253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe6ba95-cf6d-44f9-a5e6-2bb6033f9180 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:03.798790 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe6ba95-cf6d-44f9-a5e6-2bb6033f9180 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:03.800725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe6ba95-cf6d-44f9-a5e6-2bb6033f9180 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-cfe6ba95-cf6d-44f9-a5e6-2bb6033f9180 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:03.817630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:03.818840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:03.819940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:03.819940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:03.819940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:03.826873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 325/652] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:03.826873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe6ba95-cf6d-44f9-a5e6-2bb6033f9180 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:03.888509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638f65e8-7f5a-4cbf-8749-ef9c3fdf5662 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:03.888724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:03.927801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dbc395-7510-4428-852e-088266c70138 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:03.928628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dbc395-7510-4428-852e-088266c70138 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:03.929006 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dbc395-7510-4428-852e-088266c70138 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:03.929375 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dbc395-7510-4428-852e-088266c70138 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:03.961094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:03.970135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe6ba95-cf6d-44f9-a5e6-2bb6033f9180 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:04.012979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dbc395-7510-4428-852e-088266c70138 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:04.014385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dbc395-7510-4428-852e-088266c70138 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:04.022253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b6055e-e001-4ec9-9164-3b52139a02d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:04.022868 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b6055e-e001-4ec9-9164-3b52139a02d3 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:04.023147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b6055e-e001-4ec9-9164-3b52139a02d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:04.023422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b6055e-e001-4ec9-9164-3b52139a02d3 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:04.024659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b6055e-e001-4ec9-9164-3b52139a02d3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:04.025189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b6055e-e001-4ec9-9164-3b52139a02d3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:04.027392 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b6055e-e001-4ec9-9164-3b52139a02d3 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:04.041316 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 326/653] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:04.042247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b6055e-e001-4ec9-9164-3b52139a02d3 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:04.149981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 328/654] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:02:04.150805 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dbc395-7510-4428-852e-088266c70138 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:04.160587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:04.168437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b6055e-e001-4ec9-9164-3b52139a02d3 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:04.218781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:04.218979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac009f1e-913e-422c-8643-ce862ce464d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:04.219901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac009f1e-913e-422c-8643-ce862ce464d0 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:04.220195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac009f1e-913e-422c-8643-ce862ce464d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:04.220662 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac009f1e-913e-422c-8643-ce862ce464d0 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:04.221432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac009f1e-913e-422c-8643-ce862ce464d0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:04.221731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac009f1e-913e-422c-8643-ce862ce464d0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:04.224156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac009f1e-913e-422c-8643-ce862ce464d0 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:04.240501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 327/655] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:04.240501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac009f1e-913e-422c-8643-ce862ce464d0 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:04.275826 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dbc395-7510-4428-852e-088266c70138 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:04.310560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23e55a3-1d36-4f52-a800-48f2e7cee176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:04.311217 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e55a3-1d36-4f52-a800-48f2e7cee176 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:04.311487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e55a3-1d36-4f52-a800-48f2e7cee176 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:04.311725 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e55a3-1d36-4f52-a800-48f2e7cee176 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:04.320249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23e55a3-1d36-4f52-a800-48f2e7cee176 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:04.321728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23e55a3-1d36-4f52-a800-48f2e7cee176 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-e23e55a3-1d36-4f52-a800-48f2e7cee176 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:04.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:04.336301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:04.337570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:04.337570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:04.337570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:04.337570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:04.337570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:04.337570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:04.337570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:04.337570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:04.337570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 28 12:02:04.337570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:04.337570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 329/656] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:02:04 2022] GET /identity/v3/users/neutron => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:04.338168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23e55a3-1d36-4f52-a800-48f2e7cee176 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:04.350906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac009f1e-913e-422c-8643-ce862ce464d0 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:04.385596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a70fa5-7468-4bec-bdb4-b13f9fbded8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:04.386232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a70fa5-7468-4bec-bdb4-b13f9fbded8b None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:04.386507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a70fa5-7468-4bec-bdb4-b13f9fbded8b None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:04.386781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a70fa5-7468-4bec-bdb4-b13f9fbded8b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:04.397815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a70fa5-7468-4bec-bdb4-b13f9fbded8b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:04.399232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a70fa5-7468-4bec-bdb4-b13f9fbded8b None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-b2a70fa5-7468-4bec-bdb4-b13f9fbded8b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:04.413781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:04.414797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:04.415771 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:04.415771 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:04.415771 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:04.419710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 328/657] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:04.421250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4425f5f-e4b6-41f6-a3e9-be99655f6da4 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:04.421532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4425f5f-e4b6-41f6-a3e9-be99655f6da4 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:04.421801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4425f5f-e4b6-41f6-a3e9-be99655f6da4 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:04.424355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 329/658] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:04.424355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4425f5f-e4b6-41f6-a3e9-be99655f6da4 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:04.441548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:04.449428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23e55a3-1d36-4f52-a800-48f2e7cee176 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:04.464603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:04.487131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b512cd-3e13-445f-a2c8-440b788073f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:04.487763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b512cd-3e13-445f-a2c8-440b788073f2 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:04.489064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b512cd-3e13-445f-a2c8-440b788073f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:04.489064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b512cd-3e13-445f-a2c8-440b788073f2 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:04.502091 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b512cd-3e13-445f-a2c8-440b788073f2 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:04.503778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b512cd-3e13-445f-a2c8-440b788073f2 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-d9b512cd-3e13-445f-a2c8-440b788073f2 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:04.515699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:04.516861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:04.517847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:04.517847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:04.517847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:04.517847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:04.517847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:04.517847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:04.517847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:04.517847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:04.517847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 28 12:02:04.517847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:04.517847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 330/659] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:02:04 2022] GET /identity/v3/users/glance => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:04.517847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b512cd-3e13-445f-a2c8-440b788073f2 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:04.543107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4425f5f-e4b6-41f6-a3e9-be99655f6da4 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:04.578095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcf44bd-1cf3-4beb-b0a4-7cd8a4d4d45e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:04.578697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcf44bd-1cf3-4beb-b0a4-7cd8a4d4d45e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:04.578981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcf44bd-1cf3-4beb-b0a4-7cd8a4d4d45e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:04.579246 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcf44bd-1cf3-4beb-b0a4-7cd8a4d4d45e None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:04.579998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcf44bd-1cf3-4beb-b0a4-7cd8a4d4d45e None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:04.580289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcf44bd-1cf3-4beb-b0a4-7cd8a4d4d45e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:04.581734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcf44bd-1cf3-4beb-b0a4-7cd8a4d4d45e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:04.601720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 330/660] 10.222.0.26 () {56 vars in 1139 bytes} [Wed Sep 28 12:02:04 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:04.602969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d822833d-ee43-47f4-b1f6-0a44fb3161ae None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:04.603257 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d822833d-ee43-47f4-b1f6-0a44fb3161ae None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:04.603533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d822833d-ee43-47f4-b1f6-0a44fb3161ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:04.621943 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-d822833d-ee43-47f4-b1f6-0a44fb3161ae None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:04.623209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d822833d-ee43-47f4-b1f6-0a44fb3161ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:04.626094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b512cd-3e13-445f-a2c8-440b788073f2 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:04.664694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f920f68-d2c7-4a3d-915c-eae42d3bc406 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:04.665300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f920f68-d2c7-4a3d-915c-eae42d3bc406 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:04.665601 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f920f68-d2c7-4a3d-915c-eae42d3bc406 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:04.665874 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f920f68-d2c7-4a3d-915c-eae42d3bc406 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:04.666540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f920f68-d2c7-4a3d-915c-eae42d3bc406 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:04.666842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f920f68-d2c7-4a3d-915c-eae42d3bc406 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:04.668350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f920f68-d2c7-4a3d-915c-eae42d3bc406 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:04.680228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 331/661] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:04.685669 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f920f68-d2c7-4a3d-915c-eae42d3bc406 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:04.690208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:04.704690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 331/662] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:02:04.705318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d822833d-ee43-47f4-b1f6-0a44fb3161ae None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:04.721950 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:04.793965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f920f68-d2c7-4a3d-915c-eae42d3bc406 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:04.841010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d822833d-ee43-47f4-b1f6-0a44fb3161ae None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:04.847933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599abeb0-de30-44a0-b494-9c48677d1a26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:04.848908 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599abeb0-de30-44a0-b494-9c48677d1a26 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:04.849363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599abeb0-de30-44a0-b494-9c48677d1a26 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:04.849838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599abeb0-de30-44a0-b494-9c48677d1a26 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:04.850713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599abeb0-de30-44a0-b494-9c48677d1a26 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:04.851163 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599abeb0-de30-44a0-b494-9c48677d1a26 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:04.852788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599abeb0-de30-44a0-b494-9c48677d1a26 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:04.871670 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 332/663] 10.222.0.26 () {56 vars in 1137 bytes} [Wed Sep 28 12:02:04 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:04.872718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599abeb0-de30-44a0-b494-9c48677d1a26 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:04.877477 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ded93ea-757e-46fb-ac1a-6c1ebb3ab086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:04.878161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ded93ea-757e-46fb-ac1a-6c1ebb3ab086 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:04.878439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ded93ea-757e-46fb-ac1a-6c1ebb3ab086 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:04.878727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ded93ea-757e-46fb-ac1a-6c1ebb3ab086 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:04.947189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:04.949756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ded93ea-757e-46fb-ac1a-6c1ebb3ab086 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:04.951387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ded93ea-757e-46fb-ac1a-6c1ebb3ab086 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:04.971713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:04.981356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599abeb0-de30-44a0-b494-9c48677d1a26 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:05.016100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a2f59b-6d27-46d4-b414-cd71bfb7f87b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:05.016748 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a2f59b-6d27-46d4-b414-cd71bfb7f87b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:05.017027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a2f59b-6d27-46d4-b414-cd71bfb7f87b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:05.017298 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a2f59b-6d27-46d4-b414-cd71bfb7f87b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:05.025400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a2f59b-6d27-46d4-b414-cd71bfb7f87b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:05.026849 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a2f59b-6d27-46d4-b414-cd71bfb7f87b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-c1a2f59b-6d27-46d4-b414-cd71bfb7f87b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:05.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:05.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:05.042385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:05.042385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:05.042385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:05.042385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:05.042385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:05.042385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:05.042385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:05.042385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:05.042385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 333/664] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:05.042866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a2f59b-6d27-46d4-b414-cd71bfb7f87b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:05.058965 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 332/665] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:02:05.059568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ded93ea-757e-46fb-ac1a-6c1ebb3ab086 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:05.153593 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a2f59b-6d27-46d4-b414-cd71bfb7f87b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:05.174281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:05.178281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ded93ea-757e-46fb-ac1a-6c1ebb3ab086 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:05.190512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197a9e52-a27c-4269-9fd0-24dc94d46924 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:05.191118 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197a9e52-a27c-4269-9fd0-24dc94d46924 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:05.191449 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197a9e52-a27c-4269-9fd0-24dc94d46924 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:05.191678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197a9e52-a27c-4269-9fd0-24dc94d46924 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:05.215922 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:05.216979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c355503-3970-431e-8c66-3d14a6510477 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:05.217684 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c355503-3970-431e-8c66-3d14a6510477 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:05.218054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c355503-3970-431e-8c66-3d14a6510477 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:05.218415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c355503-3970-431e-8c66-3d14a6510477 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:05.229550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c355503-3970-431e-8c66-3d14a6510477 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-3c355503-3970-431e-8c66-3d14a6510477 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:05.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:05.235435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:05.236820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:05.236820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:05.236820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:05.236820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:05.236820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:05.236820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:05.236820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:05.236820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:05.236820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:05.236820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:05.236820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:05.236820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:05.236820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 333/666] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:02:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:05.236820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c355503-3970-431e-8c66-3d14a6510477 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:05.276679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197a9e52-a27c-4269-9fd0-24dc94d46924 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:05.278175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197a9e52-a27c-4269-9fd0-24dc94d46924 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:05.346904 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c355503-3970-431e-8c66-3d14a6510477 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:05.380848 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 334/667] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:02:05.381693 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197a9e52-a27c-4269-9fd0-24dc94d46924 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:05.389560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf30ecf-7f26-4907-83f3-7c337eb7af08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:05.390275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf30ecf-7f26-4907-83f3-7c337eb7af08 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:05.390658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf30ecf-7f26-4907-83f3-7c337eb7af08 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:05.391152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf30ecf-7f26-4907-83f3-7c337eb7af08 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:05.391939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf30ecf-7f26-4907-83f3-7c337eb7af08 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:05.392333 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf30ecf-7f26-4907-83f3-7c337eb7af08 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:05.393955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf30ecf-7f26-4907-83f3-7c337eb7af08 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:05.412389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 334/668] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:05.412389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf30ecf-7f26-4907-83f3-7c337eb7af08 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:05.434398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:05.438741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:05.494098 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197a9e52-a27c-4269-9fd0-24dc94d46924 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:05.525451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf30ecf-7f26-4907-83f3-7c337eb7af08 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:05.538804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62484bf7-1e96-42ad-aa67-5d0fb308c408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:05.539652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62484bf7-1e96-42ad-aa67-5d0fb308c408 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:05.540134 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62484bf7-1e96-42ad-aa67-5d0fb308c408 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:05.540642 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62484bf7-1e96-42ad-aa67-5d0fb308c408 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:05.553016 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62484bf7-1e96-42ad-aa67-5d0fb308c408 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:05.554558 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62484bf7-1e96-42ad-aa67-5d0fb308c408 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:05.564332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc56756-14da-4f1d-a516-d4f584ebe47b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:05.564998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc56756-14da-4f1d-a516-d4f584ebe47b None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:05.565347 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc56756-14da-4f1d-a516-d4f584ebe47b None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:05.565690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc56756-14da-4f1d-a516-d4f584ebe47b None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:05.566993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc56756-14da-4f1d-a516-d4f584ebe47b None admin] RBAC: Adding query filter params (name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:05.567387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc56756-14da-4f1d-a516-d4f584ebe47b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:05.568841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc56756-14da-4f1d-a516-d4f584ebe47b None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-62484bf7-1e96-42ad-aa67-5d0fb308c408 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:05.571095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:05.572019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:05.572926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:05.572926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:05.572926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:05.572926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 335/669] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:05.573175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62484bf7-1e96-42ad-aa67-5d0fb308c408 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:05.580713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 335/670] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:02:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:05.581420 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc56756-14da-4f1d-a516-d4f584ebe47b None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:05.671631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:05.679794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62484bf7-1e96-42ad-aa67-5d0fb308c408 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:05.682116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:05.693941 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc56756-14da-4f1d-a516-d4f584ebe47b None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:05.716757 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba48916-24d1-4fc9-b0ee-ea1ef2495a4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:05.717487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba48916-24d1-4fc9-b0ee-ea1ef2495a4a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:05.717835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba48916-24d1-4fc9-b0ee-ea1ef2495a4a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:05.718184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba48916-24d1-4fc9-b0ee-ea1ef2495a4a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:05.728846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba48916-24d1-4fc9-b0ee-ea1ef2495a4a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:05.728996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4fdc77-bb5e-490d-802f-2cffa91d6d4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:05.729618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4fdc77-bb5e-490d-802f-2cffa91d6d4b None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:05.729969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4fdc77-bb5e-490d-802f-2cffa91d6d4b None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:05.730313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4fdc77-bb5e-490d-802f-2cffa91d6d4b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:05.730639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba48916-24d1-4fc9-b0ee-ea1ef2495a4a None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:05.731188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4fdc77-bb5e-490d-802f-2cffa91d6d4b None admin] RBAC: Adding query filter params (role.id=ed4d262295434a72bb0a8b9a7ff12e71, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=2ba80861b08940d398a61879bb70060a) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:05.731565 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4fdc77-bb5e-490d-802f-2cffa91d6d4b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:05.733095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4fdc77-bb5e-490d-802f-2cffa91d6d4b None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:05.739943 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 336/671] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:05 2022] GET /identity/v3/role_assignments?user.id=2ba80861b08940d398a61879bb70060a&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=ed4d262295434a72bb0a8b9a7ff12e71 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:05.740632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4fdc77-bb5e-490d-802f-2cffa91d6d4b None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-cba48916-24d1-4fc9-b0ee-ea1ef2495a4a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:05.750800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:05.751829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:05.752952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:05.752952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:05.752952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:05.757380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 336/672] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:05.758367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba48916-24d1-4fc9-b0ee-ea1ef2495a4a None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:05.879711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4fdc77-bb5e-490d-802f-2cffa91d6d4b None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:05.880448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba48916-24d1-4fc9-b0ee-ea1ef2495a4a None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:05.919955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88f31bf-51ed-4602-8b19-4f001a63ee72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:05.920933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88f31bf-51ed-4602-8b19-4f001a63ee72 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:05.921398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88f31bf-51ed-4602-8b19-4f001a63ee72 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:05.921873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88f31bf-51ed-4602-8b19-4f001a63ee72 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:05.923207 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49445274-300f-4b03-b6a2-8d79eb832f84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:05.923837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49445274-300f-4b03-b6a2-8d79eb832f84 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:05.924119 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49445274-300f-4b03-b6a2-8d79eb832f84 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:05.924438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49445274-300f-4b03-b6a2-8d79eb832f84 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:05.925223 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49445274-300f-4b03-b6a2-8d79eb832f84 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:05.925562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49445274-300f-4b03-b6a2-8d79eb832f84 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:05.927690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49445274-300f-4b03-b6a2-8d79eb832f84 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:05.936185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88f31bf-51ed-4602-8b19-4f001a63ee72 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:05.938138 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88f31bf-51ed-4602-8b19-4f001a63ee72 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:05.947252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 337/673] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:05.948213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49445274-300f-4b03-b6a2-8d79eb832f84 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-e88f31bf-51ed-4602-8b19-4f001a63ee72 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:05.956935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:05.957896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:05.958764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:05.958764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:05.958764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:05.958764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 337/674] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:05.959401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88f31bf-51ed-4602-8b19-4f001a63ee72 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:05.963876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:05.975802 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:06.081976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88f31bf-51ed-4602-8b19-4f001a63ee72 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:06.082204 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49445274-300f-4b03-b6a2-8d79eb832f84 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:06.124204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba72c7b9-d589-4b61-9286-31cbd1e752a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:06.124978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba72c7b9-d589-4b61-9286-31cbd1e752a8 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:06.125258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba72c7b9-d589-4b61-9286-31cbd1e752a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:06.125531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba72c7b9-d589-4b61-9286-31cbd1e752a8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:06.134342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba72c7b9-d589-4b61-9286-31cbd1e752a8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:06.134815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c468cda6-6347-4bde-80be-1f3c5e2bdc94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:06.135494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c468cda6-6347-4bde-80be-1f3c5e2bdc94 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:06.135814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba72c7b9-d589-4b61-9286-31cbd1e752a8 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:06.135919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c468cda6-6347-4bde-80be-1f3c5e2bdc94 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:06.136162 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c468cda6-6347-4bde-80be-1f3c5e2bdc94 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:06.137016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c468cda6-6347-4bde-80be-1f3c5e2bdc94 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:06.137315 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c468cda6-6347-4bde-80be-1f3c5e2bdc94 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:06.139381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c468cda6-6347-4bde-80be-1f3c5e2bdc94 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-ba72c7b9-d589-4b61-9286-31cbd1e752a8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:06.148782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:06.149686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:06.150562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:06.150562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:06.150562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:06.150562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:06.150562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:06.150562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:06.150562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:06.150562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:06.150562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 338/675] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:06.151088 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba72c7b9-d589-4b61-9286-31cbd1e752a8 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:06.158798 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 338/676] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:06.161020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c468cda6-6347-4bde-80be-1f3c5e2bdc94 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:06.243217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:06.244745 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:06.281360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c468cda6-6347-4bde-80be-1f3c5e2bdc94 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:06.281562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba72c7b9-d589-4b61-9286-31cbd1e752a8 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:06.317434 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eaca526-f12e-4ebc-89e9-f94d1a2c0bd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:06.317674 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c69f42-26d0-4095-81aa-ae44113be8df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:06.318025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaca526-f12e-4ebc-89e9-f94d1a2c0bd8 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:06.318147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c69f42-26d0-4095-81aa-ae44113be8df None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:06.318301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaca526-f12e-4ebc-89e9-f94d1a2c0bd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:06.318396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c69f42-26d0-4095-81aa-ae44113be8df None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:06.318567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaca526-f12e-4ebc-89e9-f94d1a2c0bd8 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:06.318663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c69f42-26d0-4095-81aa-ae44113be8df None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:06.319206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eaca526-f12e-4ebc-89e9-f94d1a2c0bd8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:06.319301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c69f42-26d0-4095-81aa-ae44113be8df None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:06.319503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eaca526-f12e-4ebc-89e9-f94d1a2c0bd8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:06.319608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c69f42-26d0-4095-81aa-ae44113be8df None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:06.320909 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eaca526-f12e-4ebc-89e9-f94d1a2c0bd8 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:06.321018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c69f42-26d0-4095-81aa-ae44113be8df None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:06.333247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 339/677] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:06.333869 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eaca526-f12e-4ebc-89e9-f94d1a2c0bd8 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:06.338026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 339/678] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:06.339371 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2011041-8455-4e82-a71c-d57af3a6e256 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:06.339666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2011041-8455-4e82-a71c-d57af3a6e256 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:06.339941 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2011041-8455-4e82-a71c-d57af3a6e256 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:06.344446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 340/679] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:06.344446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2011041-8455-4e82-a71c-d57af3a6e256 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:06.439217 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eaca526-f12e-4ebc-89e9-f94d1a2c0bd8 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:06.450633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2011041-8455-4e82-a71c-d57af3a6e256 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:06.466330 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:06.476298 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424613c-96e2-426d-ad90-8fb3b9af8f37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:06.476964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424613c-96e2-426d-ad90-8fb3b9af8f37 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:06.477239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424613c-96e2-426d-ad90-8fb3b9af8f37 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:06.477507 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424613c-96e2-426d-ad90-8fb3b9af8f37 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:06.477958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:06.492005 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7424613c-96e2-426d-ad90-8fb3b9af8f37 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:06.493547 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7424613c-96e2-426d-ad90-8fb3b9af8f37 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:06.495232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c472a4-71e7-4e7c-8548-c0a7cd082d1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:06.495859 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c472a4-71e7-4e7c-8548-c0a7cd082d1e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:06.496145 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c472a4-71e7-4e7c-8548-c0a7cd082d1e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:06.496463 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c472a4-71e7-4e7c-8548-c0a7cd082d1e None admin] PATH_INFO: `/v3/users/swift` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:06.503979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c472a4-71e7-4e7c-8548-c0a7cd082d1e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-7424613c-96e2-426d-ad90-8fb3b9af8f37 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:06.505344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:06.506205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:06.507089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:06.507089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:06.507089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:06.507089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:06.507089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:06.507089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:06.507089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:06.507089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:06.508156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c472a4-71e7-4e7c-8548-c0a7cd082d1e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:06.508240 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 340/680] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:06.508240 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424613c-96e2-426d-ad90-8fb3b9af8f37 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-e9c472a4-71e7-4e7c-8548-c0a7cd082d1e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:06.522901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:06.523875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:06.525433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:06.525433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:06.525433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:06.525433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:06.525433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:06.525433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:06.525433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:06.525433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:06.525433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 28 12:02:06.525433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:06.525433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 341/681] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:02:06 2022] GET /identity/v3/users/swift => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:06.525433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78a28e8-2515-4c78-af50-f410b4373171 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:06.525433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78a28e8-2515-4c78-af50-f410b4373171 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:06.526186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78a28e8-2515-4c78-af50-f410b4373171 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:06.551746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-b78a28e8-2515-4c78-af50-f410b4373171 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:06.553147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b78a28e8-2515-4c78-af50-f410b4373171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:06.630502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424613c-96e2-426d-ad90-8fb3b9af8f37 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:06.642661 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 342/682] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:02:06.642661 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78a28e8-2515-4c78-af50-f410b4373171 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:06.665053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c158a544-d920-4489-af67-096c383b48f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:06.665651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c158a544-d920-4489-af67-096c383b48f4 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:06.665929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c158a544-d920-4489-af67-096c383b48f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:06.666197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c158a544-d920-4489-af67-096c383b48f4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:06.666921 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c158a544-d920-4489-af67-096c383b48f4 None admin] RBAC: Adding query filter params (role.id=ed4d262295434a72bb0a8b9a7ff12e71, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=0e429f061bd04c28b0c60c593b64fb8e) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:06.667223 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c158a544-d920-4489-af67-096c383b48f4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:06.668681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c158a544-d920-4489-af67-096c383b48f4 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:06.680721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 341/683] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:06 2022] GET /identity/v3/role_assignments?user.id=0e429f061bd04c28b0c60c593b64fb8e&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=ed4d262295434a72bb0a8b9a7ff12e71 => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:06.681760 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c158a544-d920-4489-af67-096c383b48f4 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:06.771063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78a28e8-2515-4c78-af50-f410b4373171 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:06.792322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:06.808205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:06.819394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b02e68-e1ee-4cd0-a91c-e4e62a4517b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:06.820022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b02e68-e1ee-4cd0-a91c-e4e62a4517b1 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:06.820317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b02e68-e1ee-4cd0-a91c-e4e62a4517b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:06.820604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b02e68-e1ee-4cd0-a91c-e4e62a4517b1 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:06.821321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b02e68-e1ee-4cd0-a91c-e4e62a4517b1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:06.821630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b02e68-e1ee-4cd0-a91c-e4e62a4517b1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:06.823099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b02e68-e1ee-4cd0-a91c-e4e62a4517b1 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:06.839026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 343/684] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:06.845178 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b02e68-e1ee-4cd0-a91c-e4e62a4517b1 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:06.940955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c158a544-d920-4489-af67-096c383b48f4 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:06.978345 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b02e68-e1ee-4cd0-a91c-e4e62a4517b1 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:06.988745 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ad9fcd-b54e-4aea-9689-45704ebf3e4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:06.988745 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ad9fcd-b54e-4aea-9689-45704ebf3e4d None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:06.989001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ad9fcd-b54e-4aea-9689-45704ebf3e4d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:06.989206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ad9fcd-b54e-4aea-9689-45704ebf3e4d None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:06.989869 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ad9fcd-b54e-4aea-9689-45704ebf3e4d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:06.990611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ad9fcd-b54e-4aea-9689-45704ebf3e4d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:06.992396 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ad9fcd-b54e-4aea-9689-45704ebf3e4d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:07.015639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 342/685] 10.222.0.26 () {56 vars in 1135 bytes} [Wed Sep 28 12:02:06 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:07.021387 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ad9fcd-b54e-4aea-9689-45704ebf3e4d None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:07.024891 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cddf43-c278-40ea-9401-4c0efd2d5252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:07.025599 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cddf43-c278-40ea-9401-4c0efd2d5252 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:07.025951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cddf43-c278-40ea-9401-4c0efd2d5252 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:07.026295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cddf43-c278-40ea-9401-4c0efd2d5252 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:07.027094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cddf43-c278-40ea-9401-4c0efd2d5252 None admin] RBAC: Adding query filter params (role.id=ed4d262295434a72bb0a8b9a7ff12e71, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=815504693bcc40b6aef06aacb2a9a35b) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:07.027470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cddf43-c278-40ea-9401-4c0efd2d5252 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:07.028997 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cddf43-c278-40ea-9401-4c0efd2d5252 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:07.035492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 344/686] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:06 2022] GET /identity/v3/role_assignments?user.id=815504693bcc40b6aef06aacb2a9a35b&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=ed4d262295434a72bb0a8b9a7ff12e71 => generated 608 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:07.036407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cddf43-c278-40ea-9401-4c0efd2d5252 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:07.121701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:07.190135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cddf43-c278-40ea-9401-4c0efd2d5252 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:07.218127 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ad9fcd-b54e-4aea-9689-45704ebf3e4d None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:07.229419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c5afdb-c9cb-4a8b-95c5-46a380af216c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:07.230356 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c5afdb-c9cb-4a8b-95c5-46a380af216c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:07.230789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c5afdb-c9cb-4a8b-95c5-46a380af216c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:07.231063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c5afdb-c9cb-4a8b-95c5-46a380af216c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:07.236827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:07.265926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c511a4b-a65a-457d-b77d-8897cff340c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:07.266556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c511a4b-a65a-457d-b77d-8897cff340c7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:07.266839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c511a4b-a65a-457d-b77d-8897cff340c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:07.267116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c511a4b-a65a-457d-b77d-8897cff340c7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:07.279112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c511a4b-a65a-457d-b77d-8897cff340c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-1c511a4b-a65a-457d-b77d-8897cff340c7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:07.284173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:07.285586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:07.286934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:07.286934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:07.286934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:07.286934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:07.286934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:07.286934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:07.286934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:07.286934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:07.286934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:07.286934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:07.286934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:07.286934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:07.286934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 343/687] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:02:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:07.297762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c511a4b-a65a-457d-b77d-8897cff340c7 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:07.317292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c5afdb-c9cb-4a8b-95c5-46a380af216c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:07.318732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c5afdb-c9cb-4a8b-95c5-46a380af216c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:07.372437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:07.454306 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c511a4b-a65a-457d-b77d-8897cff340c7 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:07.458134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 345/688] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:02:07.463643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c5afdb-c9cb-4a8b-95c5-46a380af216c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:07.502661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7500020b-2614-45ea-9a86-0ac25a0fddb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:07.503403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7500020b-2614-45ea-9a86-0ac25a0fddb7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:07.503831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7500020b-2614-45ea-9a86-0ac25a0fddb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:07.504174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7500020b-2614-45ea-9a86-0ac25a0fddb7 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:07.504949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7500020b-2614-45ea-9a86-0ac25a0fddb7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:07.505325 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7500020b-2614-45ea-9a86-0ac25a0fddb7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:07.506780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7500020b-2614-45ea-9a86-0ac25a0fddb7 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:07.516723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 344/689] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:02:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:07.530533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7500020b-2614-45ea-9a86-0ac25a0fddb7 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:07.585039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:07.617317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c5afdb-c9cb-4a8b-95c5-46a380af216c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:07.678135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985ec38e-b4f9-4d0d-8afe-7dcdc2f496f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:07.678783 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985ec38e-b4f9-4d0d-8afe-7dcdc2f496f4 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:07.679063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985ec38e-b4f9-4d0d-8afe-7dcdc2f496f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:07.679335 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985ec38e-b4f9-4d0d-8afe-7dcdc2f496f4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:07.700810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:07.703440 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7500020b-2614-45ea-9a86-0ac25a0fddb7 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:07.704563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985ec38e-b4f9-4d0d-8afe-7dcdc2f496f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:07.706051 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985ec38e-b4f9-4d0d-8afe-7dcdc2f496f4 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-985ec38e-b4f9-4d0d-8afe-7dcdc2f496f4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:07.734929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:07.735921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:07.736829 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:07.736829 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:07.736829 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:07.741686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 346/690] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:07.742773 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985ec38e-b4f9-4d0d-8afe-7dcdc2f496f4 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:07.754622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae2aeef-2732-46ef-999d-1542d350d17e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:07.755551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae2aeef-2732-46ef-999d-1542d350d17e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:07.755947 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae2aeef-2732-46ef-999d-1542d350d17e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:07.756280 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae2aeef-2732-46ef-999d-1542d350d17e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:07.776377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae2aeef-2732-46ef-999d-1542d350d17e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:07.777908 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae2aeef-2732-46ef-999d-1542d350d17e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-fae2aeef-2732-46ef-999d-1542d350d17e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:07.807734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:07.808717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:07.809686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:07.809686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:07.809686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:07.809686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 345/691] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:07.814779 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae2aeef-2732-46ef-999d-1542d350d17e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:07.877635 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985ec38e-b4f9-4d0d-8afe-7dcdc2f496f4 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:07.907468 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:07.939867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa816e20-ee0c-4244-9f3f-81a9d79b4f36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:07.944330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa816e20-ee0c-4244-9f3f-81a9d79b4f36 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:07.944330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa816e20-ee0c-4244-9f3f-81a9d79b4f36 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:07.944330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa816e20-ee0c-4244-9f3f-81a9d79b4f36 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:07.944330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa816e20-ee0c-4244-9f3f-81a9d79b4f36 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:07.944330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa816e20-ee0c-4244-9f3f-81a9d79b4f36 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:07.944330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa816e20-ee0c-4244-9f3f-81a9d79b4f36 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:07.947339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae2aeef-2732-46ef-999d-1542d350d17e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:07.960341 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 347/692] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:07.962669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa816e20-ee0c-4244-9f3f-81a9d79b4f36 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:08.021145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f101bf30-d0f6-40e0-befe-52d315bbda4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:08.024033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f101bf30-d0f6-40e0-befe-52d315bbda4c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:08.024332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f101bf30-d0f6-40e0-befe-52d315bbda4c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:08.024621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f101bf30-d0f6-40e0-befe-52d315bbda4c None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:08.025276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f101bf30-d0f6-40e0-befe-52d315bbda4c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:08.025583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f101bf30-d0f6-40e0-befe-52d315bbda4c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:08.026960 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f101bf30-d0f6-40e0-befe-52d315bbda4c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:08.048437 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 346/693] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:08.053284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f101bf30-d0f6-40e0-befe-52d315bbda4c None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:08.074373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:08.137800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa816e20-ee0c-4244-9f3f-81a9d79b4f36 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:08.173077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f101bf30-d0f6-40e0-befe-52d315bbda4c None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:08.176398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0e311b-bbd1-4fde-a280-6338a3ca7f4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:08.177527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0e311b-bbd1-4fde-a280-6338a3ca7f4c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:08.177527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0e311b-bbd1-4fde-a280-6338a3ca7f4c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:08.177731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0e311b-bbd1-4fde-a280-6338a3ca7f4c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:08.184612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0e311b-bbd1-4fde-a280-6338a3ca7f4c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:08.186225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0e311b-bbd1-4fde-a280-6338a3ca7f4c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-0d0e311b-bbd1-4fde-a280-6338a3ca7f4c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:08.198892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:08.199814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:08.200742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:08.200742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:08.200742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:08.200742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:08.200742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:08.200742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:08.200742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:08.200742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:08.205073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 348/694] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:08.206284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f0b303-ddec-4294-ba86-1aa300bbca18 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:08.206641 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f0b303-ddec-4294-ba86-1aa300bbca18 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:08.207048 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f0b303-ddec-4294-ba86-1aa300bbca18 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:08.208794 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 349/695] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:08.213283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f0b303-ddec-4294-ba86-1aa300bbca18 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:08.217594 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d7532e-31cf-4bdf-8ec4-aac58aee4636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:08.218614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d7532e-31cf-4bdf-8ec4-aac58aee4636 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:08.219383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d7532e-31cf-4bdf-8ec4-aac58aee4636 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:08.219829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d7532e-31cf-4bdf-8ec4-aac58aee4636 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:08.229226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d7532e-31cf-4bdf-8ec4-aac58aee4636 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:08.231185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d7532e-31cf-4bdf-8ec4-aac58aee4636 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-f8d7532e-31cf-4bdf-8ec4-aac58aee4636 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:08.244433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:08.245398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:08.246307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:08.246307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:08.246307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:08.246307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:08.246307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:08.246307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:08.246307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:08.246307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:08.246307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:02:08.246307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:08.248941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 347/696] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Sep 28 12:02:08 2022] GET /identity/v3/users/nova => generated 82 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:08.250385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d047384f-4639-4fe9-b773-f06a5c686872 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:08.250941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d047384f-4639-4fe9-b773-f06a5c686872 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:08.251410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d047384f-4639-4fe9-b773-f06a5c686872 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:08.268711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:08.271905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-d047384f-4639-4fe9-b773-f06a5c686872 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:08.273431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d047384f-4639-4fe9-b773-f06a5c686872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:08.325335 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f0b303-ddec-4294-ba86-1aa300bbca18 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:08.343960 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:08.360607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 348/697] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:02:08.361797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d047384f-4639-4fe9-b773-f06a5c686872 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:08.371061 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9fac3a-17a5-4a0d-b0ee-3bc9ddba3035 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:08.371912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9fac3a-17a5-4a0d-b0ee-3bc9ddba3035 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:08.372916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9fac3a-17a5-4a0d-b0ee-3bc9ddba3035 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:08.373427 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9fac3a-17a5-4a0d-b0ee-3bc9ddba3035 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:08.374324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9fac3a-17a5-4a0d-b0ee-3bc9ddba3035 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:08.374825 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9fac3a-17a5-4a0d-b0ee-3bc9ddba3035 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:08.376601 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9fac3a-17a5-4a0d-b0ee-3bc9ddba3035 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:08.392023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 350/698] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:08.402020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9fac3a-17a5-4a0d-b0ee-3bc9ddba3035 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:08.533315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d047384f-4639-4fe9-b773-f06a5c686872 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:08.582451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9fac3a-17a5-4a0d-b0ee-3bc9ddba3035 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:08.605574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1fb1d4-b54d-47dc-b804-2003649885d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:08.614779 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1fb1d4-b54d-47dc-b804-2003649885d1 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:08.615474 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1fb1d4-b54d-47dc-b804-2003649885d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:08.616067 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1fb1d4-b54d-47dc-b804-2003649885d1 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:08.617226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1fb1d4-b54d-47dc-b804-2003649885d1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:08.617827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1fb1d4-b54d-47dc-b804-2003649885d1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:08.618078 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a52edd-21c3-465e-a2b0-08bedce0c591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:08.618674 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a52edd-21c3-465e-a2b0-08bedce0c591 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:08.618952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a52edd-21c3-465e-a2b0-08bedce0c591 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:08.619227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a52edd-21c3-465e-a2b0-08bedce0c591 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:08.619949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a52edd-21c3-465e-a2b0-08bedce0c591 None admin] RBAC: Adding query filter params (role.id=ed4d262295434a72bb0a8b9a7ff12e71, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=e663aad79bb8429aa223564a15332ee2) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:08.620258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a52edd-21c3-465e-a2b0-08bedce0c591 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:08.620675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1fb1d4-b54d-47dc-b804-2003649885d1 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:08.622294 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a52edd-21c3-465e-a2b0-08bedce0c591 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:08.628667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:08.629532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 351/699] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:08 2022] GET /identity/v3/role_assignments?user.id=e663aad79bb8429aa223564a15332ee2&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=ed4d262295434a72bb0a8b9a7ff12e71 => generated 608 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:08.658250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 349/700] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Sep 28 12:02:08 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:08.664203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a52edd-21c3-465e-a2b0-08bedce0c591 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:08.734655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1fb1d4-b54d-47dc-b804-2003649885d1 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:08.734852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:08.809073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a52edd-21c3-465e-a2b0-08bedce0c591 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:08.848628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622c23ab-945f-489d-8154-1b0f9a3341e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:08.848628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c23ab-945f-489d-8154-1b0f9a3341e4 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:08.848895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c23ab-945f-489d-8154-1b0f9a3341e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:08.849127 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c23ab-945f-489d-8154-1b0f9a3341e4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:08.926113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622c23ab-945f-489d-8154-1b0f9a3341e4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:08.927752 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622c23ab-945f-489d-8154-1b0f9a3341e4 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:08.944050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1fb1d4-b54d-47dc-b804-2003649885d1 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:08.964979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:08.984619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af468643-ea1a-45b7-bfc2-58fd78340a77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:08.985255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af468643-ea1a-45b7-bfc2-58fd78340a77 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:08.985532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af468643-ea1a-45b7-bfc2-58fd78340a77 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:08.985803 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af468643-ea1a-45b7-bfc2-58fd78340a77 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:08.986498 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af468643-ea1a-45b7-bfc2-58fd78340a77 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:08.988122 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af468643-ea1a-45b7-bfc2-58fd78340a77 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-af468643-ea1a-45b7-bfc2-58fd78340a77 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:09.000123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:09.001168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:09.002073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 350/701] 10.222.0.26 () {56 vars in 1129 bytes} [Wed Sep 28 12:02:08 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:09.009680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af468643-ea1a-45b7-bfc2-58fd78340a77 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:09.035350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 352/702] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:02:09.040405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622c23ab-945f-489d-8154-1b0f9a3341e4 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:09.073431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:09.129043 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af468643-ea1a-45b7-bfc2-58fd78340a77 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:09.159685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622c23ab-945f-489d-8154-1b0f9a3341e4 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:09.168908 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c4ffdb-d17f-4bfb-97be-6db658976c94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:09.169514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c4ffdb-d17f-4bfb-97be-6db658976c94 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:09.169794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c4ffdb-d17f-4bfb-97be-6db658976c94 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:09.170229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c4ffdb-d17f-4bfb-97be-6db658976c94 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:09.170863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c4ffdb-d17f-4bfb-97be-6db658976c94 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:09.171168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c4ffdb-d17f-4bfb-97be-6db658976c94 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:09.172689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c4ffdb-d17f-4bfb-97be-6db658976c94 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:09.179154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 351/703] 10.222.0.26 () {56 vars in 1124 bytes} [Wed Sep 28 12:02:09 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:09.184352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c4ffdb-d17f-4bfb-97be-6db658976c94 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:09.198174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eed3a8-dfa6-40f4-a0fd-2f90b44760de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:09.198808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eed3a8-dfa6-40f4-a0fd-2f90b44760de None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:09.199087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eed3a8-dfa6-40f4-a0fd-2f90b44760de None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:09.199357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eed3a8-dfa6-40f4-a0fd-2f90b44760de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:09.203047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:09.213720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0eed3a8-dfa6-40f4-a0fd-2f90b44760de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:09.215568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0eed3a8-dfa6-40f4-a0fd-2f90b44760de None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-b0eed3a8-dfa6-40f4-a0fd-2f90b44760de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:09.233330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:09.234319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:09.235188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:09.235188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:09.235188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:09.235372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 353/704] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:09.241335 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eed3a8-dfa6-40f4-a0fd-2f90b44760de None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:09.356328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c4ffdb-d17f-4bfb-97be-6db658976c94 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:09.357485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eed3a8-dfa6-40f4-a0fd-2f90b44760de None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:09.395204 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc7bd83-35d5-4a99-9499-7787532ef5c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:09.395870 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc7bd83-35d5-4a99-9499-7787532ef5c1 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:09.396287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:09.397167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc7bd83-35d5-4a99-9499-7787532ef5c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:09.397442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc7bd83-35d5-4a99-9499-7787532ef5c1 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:09.398100 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc7bd83-35d5-4a99-9499-7787532ef5c1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:09.398403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc7bd83-35d5-4a99-9499-7787532ef5c1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:09.400398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc7bd83-35d5-4a99-9499-7787532ef5c1 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:09.414924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 354/705] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:09.420071 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc7bd83-35d5-4a99-9499-7787532ef5c1 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:09.436028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15e4b8e-735b-4788-94b9-61db26ce167f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:09.436668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15e4b8e-735b-4788-94b9-61db26ce167f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:09.436948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15e4b8e-735b-4788-94b9-61db26ce167f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:09.437220 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15e4b8e-735b-4788-94b9-61db26ce167f None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:09.437857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15e4b8e-735b-4788-94b9-61db26ce167f None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:09.438159 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15e4b8e-735b-4788-94b9-61db26ce167f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:09.439538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15e4b8e-735b-4788-94b9-61db26ce167f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:09.446756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 352/706] 10.222.0.26 () {56 vars in 1124 bytes} [Wed Sep 28 12:02:09 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:09.483643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e52f880-8ded-44c7-a4c3-944d36aa265c None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:09.483643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e52f880-8ded-44c7-a4c3-944d36aa265c None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:09.483643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e52f880-8ded-44c7-a4c3-944d36aa265c None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:09.483643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 353/707] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:09.494314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d618cbb-9446-4b58-bfef-a1dc9b2cbe5b None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:09.494804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d618cbb-9446-4b58-bfef-a1dc9b2cbe5b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:09.495184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d618cbb-9446-4b58-bfef-a1dc9b2cbe5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:09.512682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:09.533660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3d618cbb-9446-4b58-bfef-a1dc9b2cbe5b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:09.537519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3d618cbb-9446-4b58-bfef-a1dc9b2cbe5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:09.615169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc7bd83-35d5-4a99-9499-7787532ef5c1 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:09.633534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 354/708] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:02:09.696416 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658e4184-49a1-4f28-b0af-49aeba976d65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:09.697001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658e4184-49a1-4f28-b0af-49aeba976d65 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:09.697283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658e4184-49a1-4f28-b0af-49aeba976d65 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:09.697555 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658e4184-49a1-4f28-b0af-49aeba976d65 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:09.705646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658e4184-49a1-4f28-b0af-49aeba976d65 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:09.707964 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658e4184-49a1-4f28-b0af-49aeba976d65 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-658e4184-49a1-4f28-b0af-49aeba976d65 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:09.720553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:09.721453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:09.722902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:09.722902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:09.722902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:09.722902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:09.722902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:09.722902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:09.722902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:09.722902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:09.722902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 355/709] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:09.728106 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d618cbb-9446-4b58-bfef-a1dc9b2cbe5b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:09.814404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de9caaa-520d-41b6-b372-9673ef47c43d None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:09.814716 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de9caaa-520d-41b6-b372-9673ef47c43d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:09.814993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de9caaa-520d-41b6-b372-9673ef47c43d None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:09.817252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 356/710] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:09.822434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c2bd73-c387-4008-941f-c3c7da329b17 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:09.822717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c2bd73-c387-4008-941f-c3c7da329b17 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:09.822995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c2bd73-c387-4008-941f-c3c7da329b17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:09.842467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-56c2bd73-c387-4008-941f-c3c7da329b17 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:09.843657 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-56c2bd73-c387-4008-941f-c3c7da329b17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:09.847820 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d618cbb-9446-4b58-bfef-a1dc9b2cbe5b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:09.879703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:09.902145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf77888-2162-43dc-9ebc-3a15e77cf944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:09.903109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf77888-2162-43dc-9ebc-3a15e77cf944 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:09.903597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf77888-2162-43dc-9ebc-3a15e77cf944 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:09.904103 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf77888-2162-43dc-9ebc-3a15e77cf944 None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:09.905058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf77888-2162-43dc-9ebc-3a15e77cf944 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:09.905578 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf77888-2162-43dc-9ebc-3a15e77cf944 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:09.907284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf77888-2162-43dc-9ebc-3a15e77cf944 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:09.923811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 355/711] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:09.930304 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf77888-2162-43dc-9ebc-3a15e77cf944 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:09.979501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 357/712] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:02:09.979501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c2bd73-c387-4008-941f-c3c7da329b17 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:10.050036 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:10.122532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf77888-2162-43dc-9ebc-3a15e77cf944 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:10.140342 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c2bd73-c387-4008-941f-c3c7da329b17 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:10.173197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caaf8e6d-2477-4ff2-9c6d-c4ee0928c3fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:10.174119 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caaf8e6d-2477-4ff2-9c6d-c4ee0928c3fe None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:10.174396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caaf8e6d-2477-4ff2-9c6d-c4ee0928c3fe None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:10.174925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caaf8e6d-2477-4ff2-9c6d-c4ee0928c3fe None admin] PATH_INFO: `/v3/services/network` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:10.176036 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caaf8e6d-2477-4ff2-9c6d-c4ee0928c3fe None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:10.177470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caaf8e6d-2477-4ff2-9c6d-c4ee0928c3fe None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-caaf8e6d-2477-4ff2-9c6d-c4ee0928c3fe None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:10.188786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:10.189768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:10.190603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 358/713] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:09 2022] GET /identity/v3/services/network => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:10.197368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caaf8e6d-2477-4ff2-9c6d-c4ee0928c3fe None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:10.201165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638c0cfb-9ee5-4843-aa92-ee9ba87205b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:10.202028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638c0cfb-9ee5-4843-aa92-ee9ba87205b9 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:10.202532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638c0cfb-9ee5-4843-aa92-ee9ba87205b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:10.203011 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638c0cfb-9ee5-4843-aa92-ee9ba87205b9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:10.203931 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638c0cfb-9ee5-4843-aa92-ee9ba87205b9 None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=8bf969348e7a4158acf8fb789430b0e6) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:10.204476 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638c0cfb-9ee5-4843-aa92-ee9ba87205b9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:10.206228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638c0cfb-9ee5-4843-aa92-ee9ba87205b9 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:10.214867 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 356/714] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:09 2022] GET /identity/v3/role_assignments?user.id=8bf969348e7a4158acf8fb789430b0e6&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 259 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:10.292777 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638c0cfb-9ee5-4843-aa92-ee9ba87205b9 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:10.352015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:10.360889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caaf8e6d-2477-4ff2-9c6d-c4ee0928c3fe None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:10.364886 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:10.422449 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6987e0-1161-45bc-a7e5-43b9f4d6989e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:10.423083 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6987e0-1161-45bc-a7e5-43b9f4d6989e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:10.423365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6987e0-1161-45bc-a7e5-43b9f4d6989e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:10.423643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6987e0-1161-45bc-a7e5-43b9f4d6989e None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:10.424279 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6987e0-1161-45bc-a7e5-43b9f4d6989e None admin] RBAC: Adding query filter params (name=network) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:10.424624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6987e0-1161-45bc-a7e5-43b9f4d6989e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:10.426562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6987e0-1161-45bc-a7e5-43b9f4d6989e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:10.435818 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 359/715] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Sep 28 12:02:10 2022] GET /identity/v3/services?name=network => generated 124 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:10.436914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6987e0-1161-45bc-a7e5-43b9f4d6989e None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:10.452513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638c0cfb-9ee5-4843-aa92-ee9ba87205b9 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:10.507110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96f9f03-431b-4927-b0a4-c8320f2e9462 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:10.508085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96f9f03-431b-4927-b0a4-c8320f2e9462 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:10.508611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96f9f03-431b-4927-b0a4-c8320f2e9462 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:10.509090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96f9f03-431b-4927-b0a4-c8320f2e9462 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:10.533714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96f9f03-431b-4927-b0a4-c8320f2e9462 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:10.535742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96f9f03-431b-4927-b0a4-c8320f2e9462 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-f96f9f03-431b-4927-b0a4-c8320f2e9462 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:10.567177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:10.568149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:10.569140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:10.569140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:10.569140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:10.576779 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 357/716] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:10.584378 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96f9f03-431b-4927-b0a4-c8320f2e9462 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:10.604179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6987e0-1161-45bc-a7e5-43b9f4d6989e None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:10.642725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8a17b1-a43a-4127-83ed-1509cd81610f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:10.645042 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8a17b1-a43a-4127-83ed-1509cd81610f None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:10.646190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8a17b1-a43a-4127-83ed-1509cd81610f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:10.646845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8a17b1-a43a-4127-83ed-1509cd81610f None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:10.648812 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8a17b1-a43a-4127-83ed-1509cd81610f None admin] RBAC: Adding query filter params (type=network) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:10.650092 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8a17b1-a43a-4127-83ed-1509cd81610f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:10.652326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8a17b1-a43a-4127-83ed-1509cd81610f None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:10.658896 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 360/717] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Sep 28 12:02:10 2022] GET /identity/v3/services?type=network => generated 124 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:10.753247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:10.763203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96f9f03-431b-4927-b0a4-c8320f2e9462 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:10.803659 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452bffb0-4e78-4e43-b8b5-541e2560a841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:10.804286 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452bffb0-4e78-4e43-b8b5-541e2560a841 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:10.804592 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452bffb0-4e78-4e43-b8b5-541e2560a841 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:10.804866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452bffb0-4e78-4e43-b8b5-541e2560a841 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:10.805526 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452bffb0-4e78-4e43-b8b5-541e2560a841 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:10.805830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452bffb0-4e78-4e43-b8b5-541e2560a841 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:10.807221 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452bffb0-4e78-4e43-b8b5-541e2560a841 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:10.829432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 358/718] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:10.835024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8a17b1-a43a-4127-83ed-1509cd81610f None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:10.887320 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:11.077250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8a17b1-a43a-4127-83ed-1509cd81610f None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:11.112836 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f1dbf5-2cc8-4d7a-beb0-b53a7d1c5b26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:11.114347 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f1dbf5-2cc8-4d7a-beb0-b53a7d1c5b26 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:11.114848 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f1dbf5-2cc8-4d7a-beb0-b53a7d1c5b26 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:11.115324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f1dbf5-2cc8-4d7a-beb0-b53a7d1c5b26 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:11.116190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f1dbf5-2cc8-4d7a-beb0-b53a7d1c5b26 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:11.117846 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f1dbf5-2cc8-4d7a-beb0-b53a7d1c5b26 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:11.147896 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 361/719] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:02:10 2022] POST /identity/v3/users => generated 254 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:11.611534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5a2b74-d698-473e-a36e-5bffcd315bc9 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:11.612048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5a2b74-d698-473e-a36e-5bffcd315bc9 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:11.612126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5a2b74-d698-473e-a36e-5bffcd315bc9 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:11.614239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 359/720] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:11.620106 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a45e8fc-ebfd-4d5b-86d3-cf95f7d5ca63 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:11.621032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a45e8fc-ebfd-4d5b-86d3-cf95f7d5ca63 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:11.621560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a45e8fc-ebfd-4d5b-86d3-cf95f7d5ca63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:11.644168 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-0a45e8fc-ebfd-4d5b-86d3-cf95f7d5ca63 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:11.645806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0a45e8fc-ebfd-4d5b-86d3-cf95f7d5ca63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:11.726509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 362/721] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:02:12.041482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5a2b74-d698-473e-a36e-5bffcd315bc9 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:12.189984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5a2b74-d698-473e-a36e-5bffcd315bc9 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:12.235862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5304012b-b1d3-4750-bb1d-b0f19a568d56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:12.236528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5304012b-b1d3-4750-bb1d-b0f19a568d56 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:12.236810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5304012b-b1d3-4750-bb1d-b0f19a568d56 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:12.237080 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5304012b-b1d3-4750-bb1d-b0f19a568d56 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:12.254834 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5304012b-b1d3-4750-bb1d-b0f19a568d56 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-5304012b-b1d3-4750-bb1d-b0f19a568d56 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:12.259685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:12.260660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:12.261479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:12.261479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:12.261479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:12.261479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:12.261479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:12.261479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:12.261479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:12.261479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:12.261479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:12.261479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:12.261479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:12.261479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:12.261479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 360/722] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:02:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:12.268694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a45e8fc-ebfd-4d5b-86d3-cf95f7d5ca63 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:12.290587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:12.404056 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a45e8fc-ebfd-4d5b-86d3-cf95f7d5ca63 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:12.448378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72da0800-2b3a-4d7e-8a4d-666d6c5e48e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:12.448985 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72da0800-2b3a-4d7e-8a4d-666d6c5e48e2 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:12.449260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72da0800-2b3a-4d7e-8a4d-666d6c5e48e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:12.449527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72da0800-2b3a-4d7e-8a4d-666d6c5e48e2 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:12.450188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72da0800-2b3a-4d7e-8a4d-666d6c5e48e2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:12.450483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72da0800-2b3a-4d7e-8a4d-666d6c5e48e2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:12.451828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72da0800-2b3a-4d7e-8a4d-666d6c5e48e2 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:12.463793 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 363/723] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:02:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:12.464577 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcf0133-b447-4e5c-8fc1-b9f8a09693d9 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:12.464865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcf0133-b447-4e5c-8fc1-b9f8a09693d9 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:12.465137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcf0133-b447-4e5c-8fc1-b9f8a09693d9 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:12.466660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 361/724] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:12.472076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72da0800-2b3a-4d7e-8a4d-666d6c5e48e2 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:12.476729 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cf8dbe-7d6c-4d61-a026-d712caa521c6 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:12.477274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cf8dbe-7d6c-4d61-a026-d712caa521c6 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:12.477753 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cf8dbe-7d6c-4d61-a026-d712caa521c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:12.528987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-32cf8dbe-7d6c-4d61-a026-d712caa521c6 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:12.530182 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-32cf8dbe-7d6c-4d61-a026-d712caa521c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:12.533116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:12.573053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:12.681198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72da0800-2b3a-4d7e-8a4d-666d6c5e48e2 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:12.743058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 362/725] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:02:12.778350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6491b8a5-9682-4455-9592-d80404dd4526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:12.779362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6491b8a5-9682-4455-9592-d80404dd4526 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:12.779839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6491b8a5-9682-4455-9592-d80404dd4526 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:12.780366 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6491b8a5-9682-4455-9592-d80404dd4526 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:12.799768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6491b8a5-9682-4455-9592-d80404dd4526 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:12.801781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6491b8a5-9682-4455-9592-d80404dd4526 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-6491b8a5-9682-4455-9592-d80404dd4526 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:12.818644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:12.819596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:12.820492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:12.820492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:12.820492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:12.822671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 364/726] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 351 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:12.827900 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cf8dbe-7d6c-4d61-a026-d712caa521c6 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:12.888353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4985b1-9ca9-4187-937d-c4ca7822fdb8 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:12.888353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4985b1-9ca9-4187-937d-c4ca7822fdb8 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:12.888353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4985b1-9ca9-4187-937d-c4ca7822fdb8 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:12.890594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 365/727] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:12.896956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9318c01c-0725-4837-9336-29aad2a148cf None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:12.897243 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9318c01c-0725-4837-9336-29aad2a148cf None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:12.897517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9318c01c-0725-4837-9336-29aad2a148cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:12.936466 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-9318c01c-0725-4837-9336-29aad2a148cf None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:12.937595 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-9318c01c-0725-4837-9336-29aad2a148cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:12.992340 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cf8dbe-7d6c-4d61-a026-d712caa521c6 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:13.049787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fb50f7-cf9a-4e62-a33e-71eec4182594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:13.054354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fb50f7-cf9a-4e62-a33e-71eec4182594 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:13.054652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fb50f7-cf9a-4e62-a33e-71eec4182594 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:13.054929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fb50f7-cf9a-4e62-a33e-71eec4182594 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:13.055590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fb50f7-cf9a-4e62-a33e-71eec4182594 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:13.055896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fb50f7-cf9a-4e62-a33e-71eec4182594 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:13.057319 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fb50f7-cf9a-4e62-a33e-71eec4182594 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:13.089771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:13.113025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 363/728] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:13.113951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fb50f7-cf9a-4e62-a33e-71eec4182594 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:13.139174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 366/729] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 28 12:02:13.165111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9318c01c-0725-4837-9336-29aad2a148cf None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:13.300120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:13.313338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fb50f7-cf9a-4e62-a33e-71eec4182594 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:13.377035 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9318c01c-0725-4837-9336-29aad2a148cf None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:13.386511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b51d90a-edba-4cf5-8282-88caa495a207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:13.387150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b51d90a-edba-4cf5-8282-88caa495a207 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:13.387426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b51d90a-edba-4cf5-8282-88caa495a207 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:13.387696 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b51d90a-edba-4cf5-8282-88caa495a207 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:13.398558 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b51d90a-edba-4cf5-8282-88caa495a207 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:13.400018 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b51d90a-edba-4cf5-8282-88caa495a207 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-9b51d90a-edba-4cf5-8282-88caa495a207 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:13.420393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:13.421339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:13.422249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:13.422249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:13.422249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:13.422249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:13.422249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:13.422249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:13.422249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:13.422249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:13.422249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 28 12:02:13.422249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:13.422249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 364/730] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:02:13 2022] GET /identity/v3/users/swift => generated 83 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:13.423413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318eaa07-c7f6-457e-b029-fd23b7661d3d None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:13.423521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318eaa07-c7f6-457e-b029-fd23b7661d3d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:13.424337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318eaa07-c7f6-457e-b029-fd23b7661d3d None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:13.428418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 365/731] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:13.429187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318eaa07-c7f6-457e-b029-fd23b7661d3d None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:13.437795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e93de4-30a8-45ad-9b0c-07b7d0b0b4a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:13.438667 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e93de4-30a8-45ad-9b0c-07b7d0b0b4a2 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:13.438954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e93de4-30a8-45ad-9b0c-07b7d0b0b4a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:13.439226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e93de4-30a8-45ad-9b0c-07b7d0b0b4a2 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:13.439897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e93de4-30a8-45ad-9b0c-07b7d0b0b4a2 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:13.441336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e93de4-30a8-45ad-9b0c-07b7d0b0b4a2 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:13.460394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 367/732] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Sep 28 12:02:13 2022] POST /identity/v3/services => generated 251 bytes in 295 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:13.460579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d4f1b6-38a9-40f2-a213-35749a0e398c None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:13.461016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d4f1b6-38a9-40f2-a213-35749a0e398c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:13.461016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d4f1b6-38a9-40f2-a213-35749a0e398c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:13.491069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-50d4f1b6-38a9-40f2-a213-35749a0e398c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:13.492366 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-50d4f1b6-38a9-40f2-a213-35749a0e398c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:13.512390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:13.580399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318eaa07-c7f6-457e-b029-fd23b7661d3d None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:13.593913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 368/733] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:13 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 28 12:02:13.594802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d4f1b6-38a9-40f2-a213-35749a0e398c None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:13.650444 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405293f0-b4cc-4a06-a15f-9faac2169ca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:13.651064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405293f0-b4cc-4a06-a15f-9faac2169ca4 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:13.651346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405293f0-b4cc-4a06-a15f-9faac2169ca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:13.651620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405293f0-b4cc-4a06-a15f-9faac2169ca4 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:13.652378 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405293f0-b4cc-4a06-a15f-9faac2169ca4 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:13.652688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405293f0-b4cc-4a06-a15f-9faac2169ca4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:13.653441 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:13.654536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405293f0-b4cc-4a06-a15f-9faac2169ca4 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:13.690076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 366/734] 10.222.0.26 () {56 vars in 1135 bytes} [Wed Sep 28 12:02:13 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:13.696411 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405293f0-b4cc-4a06-a15f-9faac2169ca4 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:13.726876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d4f1b6-38a9-40f2-a213-35749a0e398c None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:13.772638 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fed0b8d-79e5-42e1-b8d0-cd607bbd8c49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:13.773708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fed0b8d-79e5-42e1-b8d0-cd607bbd8c49 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:13.774928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fed0b8d-79e5-42e1-b8d0-cd607bbd8c49 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:13.775434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fed0b8d-79e5-42e1-b8d0-cd607bbd8c49 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:13.788514 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fed0b8d-79e5-42e1-b8d0-cd607bbd8c49 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-7fed0b8d-79e5-42e1-b8d0-cd607bbd8c49 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:13.794737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:13.795682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:13.796587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:13.796587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:13.796587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:13.796587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:13.796587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:13.796587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:13.796587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:13.796587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:13.796587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:13.796587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:13.796587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:13.796587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:13.800245 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 369/735] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:02:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:13.809997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405293f0-b4cc-4a06-a15f-9faac2169ca4 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:13.814574 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fed0b8d-79e5-42e1-b8d0-cd607bbd8c49 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:13.834707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:13.864516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0315bd-1a37-42d0-87ed-6cc40d79a4f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:13.865123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0315bd-1a37-42d0-87ed-6cc40d79a4f5 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:13.868776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0315bd-1a37-42d0-87ed-6cc40d79a4f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:13.869068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0315bd-1a37-42d0-87ed-6cc40d79a4f5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:13.969922 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0315bd-1a37-42d0-87ed-6cc40d79a4f5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:13.973149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0315bd-1a37-42d0-87ed-6cc40d79a4f5 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:13.977977 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fed0b8d-79e5-42e1-b8d0-cd607bbd8c49 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:13.989003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:14.027917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817508fe-5489-4c72-89b7-a40c7e198bc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:14.028824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817508fe-5489-4c72-89b7-a40c7e198bc6 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:14.029103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817508fe-5489-4c72-89b7-a40c7e198bc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:14.029372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817508fe-5489-4c72-89b7-a40c7e198bc6 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:14.030307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817508fe-5489-4c72-89b7-a40c7e198bc6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:14.031009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817508fe-5489-4c72-89b7-a40c7e198bc6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:14.032404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817508fe-5489-4c72-89b7-a40c7e198bc6 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:14.045427 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 370/736] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:02:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:14.046237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817508fe-5489-4c72-89b7-a40c7e198bc6 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:14.096195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 367/737] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:13 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 28 12:02:14.097774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0315bd-1a37-42d0-87ed-6cc40d79a4f5 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:14.113289 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:14.162813 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817508fe-5489-4c72-89b7-a40c7e198bc6 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:14.196908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec33089-96fe-47fc-8c5e-38e4a20a3cb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:14.197499 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec33089-96fe-47fc-8c5e-38e4a20a3cb4 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:14.197769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec33089-96fe-47fc-8c5e-38e4a20a3cb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:14.198052 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec33089-96fe-47fc-8c5e-38e4a20a3cb4 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:14.198689 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec33089-96fe-47fc-8c5e-38e4a20a3cb4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:14.200081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec33089-96fe-47fc-8c5e-38e4a20a3cb4 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:14.204622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0315bd-1a37-42d0-87ed-6cc40d79a4f5 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:14.215435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 371/738] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Sep 28 12:02:14 2022] POST /identity/v3/services => generated 240 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:14.215435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b52dc85-c8cf-4167-a3fc-3a5fef6df6d5 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:14.215435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b52dc85-c8cf-4167-a3fc-3a5fef6df6d5 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:14.215435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b52dc85-c8cf-4167-a3fc-3a5fef6df6d5 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:14.215435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 372/739] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:14.215435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b52dc85-c8cf-4167-a3fc-3a5fef6df6d5 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:14.266963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:14.270231 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2de8e1e-d0fd-46d6-999d-4276ed1132e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:14.270944 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de8e1e-d0fd-46d6-999d-4276ed1132e7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:14.271226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de8e1e-d0fd-46d6-999d-4276ed1132e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:14.271538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de8e1e-d0fd-46d6-999d-4276ed1132e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:14.287162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2de8e1e-d0fd-46d6-999d-4276ed1132e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:14.288731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2de8e1e-d0fd-46d6-999d-4276ed1132e7 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-d2de8e1e-d0fd-46d6-999d-4276ed1132e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:14.310951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:14.311943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:14.313170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:14.313170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:14.313170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:14.313170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 368/740] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:14.316331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd80208-77e4-4db6-8c5a-4f47347951b6 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:14.316331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd80208-77e4-4db6-8c5a-4f47347951b6 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:14.316331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd80208-77e4-4db6-8c5a-4f47347951b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:14.337737 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-7dd80208-77e4-4db6-8c5a-4f47347951b6 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:14.339369 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-7dd80208-77e4-4db6-8c5a-4f47347951b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:14.366961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b52dc85-c8cf-4167-a3fc-3a5fef6df6d5 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:14.407112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d24b6cd-7143-4f59-b19d-4ef8f7d35ce1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:14.407742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d24b6cd-7143-4f59-b19d-4ef8f7d35ce1 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:14.408022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d24b6cd-7143-4f59-b19d-4ef8f7d35ce1 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:14.408394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d24b6cd-7143-4f59-b19d-4ef8f7d35ce1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:14.417984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:14.419506 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d24b6cd-7143-4f59-b19d-4ef8f7d35ce1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:14.421270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d24b6cd-7143-4f59-b19d-4ef8f7d35ce1 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:14.424301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 369/741] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:14 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:14.426117 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd80208-77e4-4db6-8c5a-4f47347951b6 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-0d24b6cd-7143-4f59-b19d-4ef8f7d35ce1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:14.444233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:14.445292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:14.446169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:14.446169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:14.446169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:14.448147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 373/742] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:14.453889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d24b6cd-7143-4f59-b19d-4ef8f7d35ce1 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:14.556423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:14.562523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd80208-77e4-4db6-8c5a-4f47347951b6 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:14.596299 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d24b6cd-7143-4f59-b19d-4ef8f7d35ce1 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:14.641113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cbc890-72fa-4c13-bc0f-c57f2852a45b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:14.641657 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cbc890-72fa-4c13-bc0f-c57f2852a45b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:14.641942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cbc890-72fa-4c13-bc0f-c57f2852a45b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:14.642215 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cbc890-72fa-4c13-bc0f-c57f2852a45b None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:14.642872 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cbc890-72fa-4c13-bc0f-c57f2852a45b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:14.643180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cbc890-72fa-4c13-bc0f-c57f2852a45b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:14.644646 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cbc890-72fa-4c13-bc0f-c57f2852a45b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:14.654807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1cacf6-c5b3-46f1-8c2a-649ed9da14dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:14.655418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1cacf6-c5b3-46f1-8c2a-649ed9da14dc None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:14.655693 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1cacf6-c5b3-46f1-8c2a-649ed9da14dc None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:14.655965 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1cacf6-c5b3-46f1-8c2a-649ed9da14dc None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:14.656640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1cacf6-c5b3-46f1-8c2a-649ed9da14dc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:14.656938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1cacf6-c5b3-46f1-8c2a-649ed9da14dc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:14.658278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1cacf6-c5b3-46f1-8c2a-649ed9da14dc None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:14.666369 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 370/743] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:14.673614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cbc890-72fa-4c13-bc0f-c57f2852a45b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:14.693723 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 374/744] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:14.694699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1cacf6-c5b3-46f1-8c2a-649ed9da14dc None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:14.771058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:14.835767 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cbc890-72fa-4c13-bc0f-c57f2852a45b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:14.840703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1cacf6-c5b3-46f1-8c2a-649ed9da14dc None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:14.881314 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cb2b19-e7a3-4030-9e93-8b30dcaa03f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:14.882063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cb2b19-e7a3-4030-9e93-8b30dcaa03f5 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:14.882410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cb2b19-e7a3-4030-9e93-8b30dcaa03f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:14.882755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cb2b19-e7a3-4030-9e93-8b30dcaa03f5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:14.883907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57192950-bf6d-45c3-9e81-f2de7dc85e50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:14.885216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57192950-bf6d-45c3-9e81-f2de7dc85e50 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:14.885685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57192950-bf6d-45c3-9e81-f2de7dc85e50 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:14.886548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57192950-bf6d-45c3-9e81-f2de7dc85e50 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:14.891281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cb2b19-e7a3-4030-9e93-8b30dcaa03f5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:14.892821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cb2b19-e7a3-4030-9e93-8b30dcaa03f5 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:14.896001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:14.897773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57192950-bf6d-45c3-9e81-f2de7dc85e50 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:14.899490 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57192950-bf6d-45c3-9e81-f2de7dc85e50 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-76cb2b19-e7a3-4030-9e93-8b30dcaa03f5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:14.908094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:14.909018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:14.910451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:14.910451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:14.910451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:14.910451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:14.910451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:14.910451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:14.910451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:14.910451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:14.912909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 375/745] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:14.913867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cb2b19-e7a3-4030-9e93-8b30dcaa03f5 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-57192950-bf6d-45c3-9e81-f2de7dc85e50 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:14.918383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:14.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:14.920235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:14.920235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:14.920235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:14.920235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:14.920235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:14.920235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:14.920235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:14.920235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:14.920235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:02:14.920235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:14.920235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 371/746] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Sep 28 12:02:14 2022] GET /identity/v3/users/nova => generated 82 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:14.920864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57192950-bf6d-45c3-9e81-f2de7dc85e50 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:15.031613 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57192950-bf6d-45c3-9e81-f2de7dc85e50 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:15.036668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cb2b19-e7a3-4030-9e93-8b30dcaa03f5 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:15.046200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:15.069307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0991d03a-dbcf-4f9e-8738-344bfecb05d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:15.070292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0991d03a-dbcf-4f9e-8738-344bfecb05d1 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:15.070855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0991d03a-dbcf-4f9e-8738-344bfecb05d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:15.071332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0991d03a-dbcf-4f9e-8738-344bfecb05d1 None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:15.072225 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0991d03a-dbcf-4f9e-8738-344bfecb05d1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:15.072843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0991d03a-dbcf-4f9e-8738-344bfecb05d1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:15.074267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3a9c50-8e2f-46a3-8cfc-61c9a88c6df9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:15.074970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0991d03a-dbcf-4f9e-8738-344bfecb05d1 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:15.075102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a9c50-8e2f-46a3-8cfc-61c9a88c6df9 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:15.075313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a9c50-8e2f-46a3-8cfc-61c9a88c6df9 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:15.075590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a9c50-8e2f-46a3-8cfc-61c9a88c6df9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:15.086245 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3a9c50-8e2f-46a3-8cfc-61c9a88c6df9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-3f3a9c50-8e2f-46a3-8cfc-61c9a88c6df9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:15.094307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:15.095236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:15.096099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:15.096099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:15.096099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:15.096099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:15.096099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:15.096099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:15.096099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:15.096099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:15.096099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:15.096099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:15.096099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:15.096099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:15.096099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 376/748] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:02:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:15.096099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3a9c50-8e2f-46a3-8cfc-61c9a88c6df9 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:15.096967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 372/748] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:15.098838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0991d03a-dbcf-4f9e-8738-344bfecb05d1 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:15.167439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:15.208129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0991d03a-dbcf-4f9e-8738-344bfecb05d1 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:15.217978 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3a9c50-8e2f-46a3-8cfc-61c9a88c6df9 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:15.244911 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5846d8-9008-403a-bfda-b5abfdcea455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:15.245877 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5846d8-9008-403a-bfda-b5abfdcea455 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:15.246360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5846d8-9008-403a-bfda-b5abfdcea455 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:15.246831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5846d8-9008-403a-bfda-b5abfdcea455 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:15.247757 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5846d8-9008-403a-bfda-b5abfdcea455 None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=e663aad79bb8429aa223564a15332ee2) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:15.248278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5846d8-9008-403a-bfda-b5abfdcea455 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:15.249940 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5846d8-9008-403a-bfda-b5abfdcea455 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:15.255801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1d4c62-3646-4c3e-bd13-ea49dd106b29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:15.256355 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 373/749] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:15 2022] GET /identity/v3/role_assignments?user.id=e663aad79bb8429aa223564a15332ee2&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 259 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:15.257371 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5846d8-9008-403a-bfda-b5abfdcea455 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:15.260625 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1d4c62-3646-4c3e-bd13-ea49dd106b29 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:15.261202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1d4c62-3646-4c3e-bd13-ea49dd106b29 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:15.261679 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1d4c62-3646-4c3e-bd13-ea49dd106b29 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:15.262680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1d4c62-3646-4c3e-bd13-ea49dd106b29 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:15.263198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1d4c62-3646-4c3e-bd13-ea49dd106b29 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:15.267637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1d4c62-3646-4c3e-bd13-ea49dd106b29 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:15.309916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 377/750] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Sep 28 12:02:15 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:15.316992 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1d4c62-3646-4c3e-bd13-ea49dd106b29 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:15.344200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:15.426489 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5846d8-9008-403a-bfda-b5abfdcea455 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:15.468573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db655c37-0877-4a31-afe5-268f2894eee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:15.469192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db655c37-0877-4a31-afe5-268f2894eee9 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:15.469471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db655c37-0877-4a31-afe5-268f2894eee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:15.469743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db655c37-0877-4a31-afe5-268f2894eee9 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:15.470975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db655c37-0877-4a31-afe5-268f2894eee9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:15.471282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db655c37-0877-4a31-afe5-268f2894eee9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:15.472914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db655c37-0877-4a31-afe5-268f2894eee9 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:15.483653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 374/751] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:02:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:15.490031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db655c37-0877-4a31-afe5-268f2894eee9 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:15.544488 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1d4c62-3646-4c3e-bd13-ea49dd106b29 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:15.565126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:15.600342 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cbe363-ffed-4df0-bc62-0287daa5e790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:15.601411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cbe363-ffed-4df0-bc62-0287daa5e790 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:15.601945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cbe363-ffed-4df0-bc62-0287daa5e790 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:15.602440 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cbe363-ffed-4df0-bc62-0287daa5e790 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:15.618400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:15.623578 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db655c37-0877-4a31-afe5-268f2894eee9 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:15.624848 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cbe363-ffed-4df0-bc62-0287daa5e790 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:15.626573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cbe363-ffed-4df0-bc62-0287daa5e790 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-08cbe363-ffed-4df0-bc62-0287daa5e790 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:15.653434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:15.654423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:15.655298 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:15.655298 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:15.655298 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:15.661006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 378/752] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 345 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:15.668147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cbe363-ffed-4df0-bc62-0287daa5e790 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:15.700154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86323c1a-ddd7-4373-87a7-c5f727921c07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:15.701157 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86323c1a-ddd7-4373-87a7-c5f727921c07 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:15.701652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86323c1a-ddd7-4373-87a7-c5f727921c07 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:15.702123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86323c1a-ddd7-4373-87a7-c5f727921c07 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:15.717555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86323c1a-ddd7-4373-87a7-c5f727921c07 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:15.719429 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86323c1a-ddd7-4373-87a7-c5f727921c07 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-86323c1a-ddd7-4373-87a7-c5f727921c07 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:15.735756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:15.736785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:15.737685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:15.737685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:15.737685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:15.740260 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 375/753] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:15.749742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86323c1a-ddd7-4373-87a7-c5f727921c07 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:15.878753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cbe363-ffed-4df0-bc62-0287daa5e790 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:15.917598 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c74cbeb-b7a1-49b6-81eb-e973f5aff773 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:15.918315 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c74cbeb-b7a1-49b6-81eb-e973f5aff773 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:15.918609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c74cbeb-b7a1-49b6-81eb-e973f5aff773 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:15.918954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c74cbeb-b7a1-49b6-81eb-e973f5aff773 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:15.919607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c74cbeb-b7a1-49b6-81eb-e973f5aff773 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:15.920096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c74cbeb-b7a1-49b6-81eb-e973f5aff773 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:15.928368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c74cbeb-b7a1-49b6-81eb-e973f5aff773 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:15.929642 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86323c1a-ddd7-4373-87a7-c5f727921c07 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:15.942591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 379/754] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:15.944114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbe027c-475b-45bc-87c7-f6e6a1614364 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:15.944646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbe027c-475b-45bc-87c7-f6e6a1614364 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:15.945094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbe027c-475b-45bc-87c7-f6e6a1614364 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:15.946454 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 380/755] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:15.952598 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dce7c0-428c-43fa-a79d-d12f1513d9ca None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:15.953342 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dce7c0-428c-43fa-a79d-d12f1513d9ca None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:15.953808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dce7c0-428c-43fa-a79d-d12f1513d9ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:15.974196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-97dce7c0-428c-43fa-a79d-d12f1513d9ca None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:15.975310 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694f052d-c660-4da2-a9e5-9bbd96b7a895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:15.975484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-97dce7c0-428c-43fa-a79d-d12f1513d9ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:15.976268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694f052d-c660-4da2-a9e5-9bbd96b7a895 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:15.976789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694f052d-c660-4da2-a9e5-9bbd96b7a895 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:15.977204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694f052d-c660-4da2-a9e5-9bbd96b7a895 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:15.978106 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694f052d-c660-4da2-a9e5-9bbd96b7a895 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:15.978651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694f052d-c660-4da2-a9e5-9bbd96b7a895 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:15.980075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694f052d-c660-4da2-a9e5-9bbd96b7a895 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:15.993721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 376/756] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:15.994356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694f052d-c660-4da2-a9e5-9bbd96b7a895 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:16.003258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:16.008621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:16.067844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 381/757] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:15 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:16.069038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dce7c0-428c-43fa-a79d-d12f1513d9ca None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:16.183446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dce7c0-428c-43fa-a79d-d12f1513d9ca None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:16.197642 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694f052d-c660-4da2-a9e5-9bbd96b7a895 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:16.245851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b419ce1-2540-47bc-bc5b-1bb20b69cf96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:16.246403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b419ce1-2540-47bc-bc5b-1bb20b69cf96 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:16.246675 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b419ce1-2540-47bc-bc5b-1bb20b69cf96 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:16.246944 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b419ce1-2540-47bc-bc5b-1bb20b69cf96 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:16.260206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b419ce1-2540-47bc-bc5b-1bb20b69cf96 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:16.261826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b419ce1-2540-47bc-bc5b-1bb20b69cf96 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-0b419ce1-2540-47bc-bc5b-1bb20b69cf96 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:16.276777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:16.277888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:16.278835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:16.278835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:16.278835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:16.278835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:16.278835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:16.278835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:16.278835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:16.278835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:16.278835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 28 12:02:16.278835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:16.280073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 382/758] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Sep 28 12:02:16 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:16.285129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b419ce1-2540-47bc-bc5b-1bb20b69cf96 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:16.299859 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e80738-5198-469e-95e3-a910b63b61b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:16.301107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e80738-5198-469e-95e3-a910b63b61b0 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:16.301388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e80738-5198-469e-95e3-a910b63b61b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:16.301664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e80738-5198-469e-95e3-a910b63b61b0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:16.310878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:16.321300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e80738-5198-469e-95e3-a910b63b61b0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:16.322789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e80738-5198-469e-95e3-a910b63b61b0 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-f4e80738-5198-469e-95e3-a910b63b61b0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:16.342416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:16.344581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:16.346688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:16.346688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:16.346688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:16.346688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:16.346688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:16.346688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:16.346688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:16.346688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:16.349451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 377/759] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 355 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:16.357267 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e80738-5198-469e-95e3-a910b63b61b0 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:16.402810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b419ce1-2540-47bc-bc5b-1bb20b69cf96 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:16.445174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19c7161-50df-4cd0-90cf-3f3adb35e56b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:16.445779 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19c7161-50df-4cd0-90cf-3f3adb35e56b None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:16.446052 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19c7161-50df-4cd0-90cf-3f3adb35e56b None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:16.446318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19c7161-50df-4cd0-90cf-3f3adb35e56b None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:16.447048 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19c7161-50df-4cd0-90cf-3f3adb35e56b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:16.447348 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19c7161-50df-4cd0-90cf-3f3adb35e56b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:16.448811 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19c7161-50df-4cd0-90cf-3f3adb35e56b None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:16.450139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:16.468770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 383/760] 10.222.0.26 () {56 vars in 1149 bytes} [Wed Sep 28 12:02:16 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:16.474044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19c7161-50df-4cd0-90cf-3f3adb35e56b None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:16.501504 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e80738-5198-469e-95e3-a910b63b61b0 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:16.541367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191c1ac4-e45b-42bf-a888-91aa28f9c419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:16.541969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191c1ac4-e45b-42bf-a888-91aa28f9c419 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:16.542242 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191c1ac4-e45b-42bf-a888-91aa28f9c419 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:16.542507 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191c1ac4-e45b-42bf-a888-91aa28f9c419 None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:16.543198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191c1ac4-e45b-42bf-a888-91aa28f9c419 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:16.543498 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191c1ac4-e45b-42bf-a888-91aa28f9c419 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:16.544947 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191c1ac4-e45b-42bf-a888-91aa28f9c419 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:16.557474 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 378/761] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:16.558409 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191c1ac4-e45b-42bf-a888-91aa28f9c419 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:16.576753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:16.607343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19c7161-50df-4cd0-90cf-3f3adb35e56b None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:16.650177 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65289c94-1856-47ec-a904-0d3856c3c1bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:16.650813 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65289c94-1856-47ec-a904-0d3856c3c1bf None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:16.651117 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65289c94-1856-47ec-a904-0d3856c3c1bf None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:16.651629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65289c94-1856-47ec-a904-0d3856c3c1bf None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:16.652227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65289c94-1856-47ec-a904-0d3856c3c1bf None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:16.654303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65289c94-1856-47ec-a904-0d3856c3c1bf None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-65289c94-1856-47ec-a904-0d3856c3c1bf None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:16.667062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:16.668047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:16.668924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 384/762] 10.222.0.26 () {56 vars in 1129 bytes} [Wed Sep 28 12:02:16 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:16.669700 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65289c94-1856-47ec-a904-0d3856c3c1bf None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:16.704255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191c1ac4-e45b-42bf-a888-91aa28f9c419 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:16.734878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:16.742599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627d7b46-5db7-4a48-a562-0bfd6f0dcd16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:16.743195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627d7b46-5db7-4a48-a562-0bfd6f0dcd16 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:16.743482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627d7b46-5db7-4a48-a562-0bfd6f0dcd16 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:16.743757 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627d7b46-5db7-4a48-a562-0bfd6f0dcd16 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:16.794088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65289c94-1856-47ec-a904-0d3856c3c1bf None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:16.815109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627d7b46-5db7-4a48-a562-0bfd6f0dcd16 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:16.816655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627d7b46-5db7-4a48-a562-0bfd6f0dcd16 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:16.829660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2c3394-ec31-4fda-8074-96c8dce9bacc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:16.830319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2c3394-ec31-4fda-8074-96c8dce9bacc None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:16.830601 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2c3394-ec31-4fda-8074-96c8dce9bacc None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:16.830880 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2c3394-ec31-4fda-8074-96c8dce9bacc None admin] PATH_INFO: `/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/8bf969348e7a4158acf8fb789430b0e6/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:16.833353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e2c3394-ec31-4fda-8074-96c8dce9bacc None admin] RBAC: Authorizing `identity:create_grant(project_id=8695d562e3e24c7c9300cf3a23cd7502, user_id=8bf969348e7a4158acf8fb789430b0e6, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:16.834792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e2c3394-ec31-4fda-8074-96c8dce9bacc None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:16.848632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 385/763] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:02:16 2022] PUT /identity/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/8bf969348e7a4158acf8fb789430b0e6/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:02:16.849421 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2c3394-ec31-4fda-8074-96c8dce9bacc None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:16.851890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:16.951390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 379/764] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:16 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:02:16.951390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f4e707-c11a-430a-a854-7f72890887e4 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:16.951390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f4e707-c11a-430a-a854-7f72890887e4 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:16.951390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f4e707-c11a-430a-a854-7f72890887e4 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:16.952252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 380/765] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:16.957184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f4e707-c11a-430a-a854-7f72890887e4 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:16.977860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2c3394-ec31-4fda-8074-96c8dce9bacc None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:17.006024 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:17.023870 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb26bcd4-69b4-41ff-80a3-1b0a43a80827 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:17.024664 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb26bcd4-69b4-41ff-80a3-1b0a43a80827 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:17.025017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb26bcd4-69b4-41ff-80a3-1b0a43a80827 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:17.025404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb26bcd4-69b4-41ff-80a3-1b0a43a80827 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:17.026270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb26bcd4-69b4-41ff-80a3-1b0a43a80827 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:17.026580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb26bcd4-69b4-41ff-80a3-1b0a43a80827 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:17.028042 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb26bcd4-69b4-41ff-80a3-1b0a43a80827 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:17.039089 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 386/766] 10.222.0.26 () {56 vars in 1124 bytes} [Wed Sep 28 12:02:16 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:17.040814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752e8445-1f33-4a3b-b96f-7ac42fe43bc3 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:17.041173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752e8445-1f33-4a3b-b96f-7ac42fe43bc3 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:17.041528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752e8445-1f33-4a3b-b96f-7ac42fe43bc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:17.063952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-752e8445-1f33-4a3b-b96f-7ac42fe43bc3 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:17.065112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-752e8445-1f33-4a3b-b96f-7ac42fe43bc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:17.104279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f4e707-c11a-430a-a854-7f72890887e4 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:17.140719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:17.150476 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 387/767] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:17 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:17.151283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752e8445-1f33-4a3b-b96f-7ac42fe43bc3 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:17.160398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ea9286-9877-434e-bc79-7c16a47276a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:17.162283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ea9286-9877-434e-bc79-7c16a47276a3 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:17.162778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ea9286-9877-434e-bc79-7c16a47276a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:17.163303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ea9286-9877-434e-bc79-7c16a47276a3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:17.175914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ea9286-9877-434e-bc79-7c16a47276a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:17.178131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ea9286-9877-434e-bc79-7c16a47276a3 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-01ea9286-9877-434e-bc79-7c16a47276a3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:17.196192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:17.197285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:17.198181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:17.198181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:17.198181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:17.200631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 381/768] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:17.206093 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ea9286-9877-434e-bc79-7c16a47276a3 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:17.282856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752e8445-1f33-4a3b-b96f-7ac42fe43bc3 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:17.318573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:17.319506 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee5df47-2b6e-4431-a544-853fc1593a70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:17.320442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee5df47-2b6e-4431-a544-853fc1593a70 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:17.320838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee5df47-2b6e-4431-a544-853fc1593a70 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:17.321261 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee5df47-2b6e-4431-a544-853fc1593a70 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:17.322006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee5df47-2b6e-4431-a544-853fc1593a70 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:17.322434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee5df47-2b6e-4431-a544-853fc1593a70 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:17.323986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee5df47-2b6e-4431-a544-853fc1593a70 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:17.329553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 388/769] 10.222.0.26 () {56 vars in 1124 bytes} [Wed Sep 28 12:02:17 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:17.337535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee5df47-2b6e-4431-a544-853fc1593a70 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:17.352198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ea9286-9877-434e-bc79-7c16a47276a3 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:17.405218 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905fbc01-a22f-4c3b-999b-ac3cbd69662e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:17.406247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905fbc01-a22f-4c3b-999b-ac3cbd69662e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:17.408208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905fbc01-a22f-4c3b-999b-ac3cbd69662e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:17.408766 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905fbc01-a22f-4c3b-999b-ac3cbd69662e None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:17.409522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905fbc01-a22f-4c3b-999b-ac3cbd69662e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:17.409974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905fbc01-a22f-4c3b-999b-ac3cbd69662e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:17.411431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905fbc01-a22f-4c3b-999b-ac3cbd69662e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:17.430197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 382/770] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:17.435631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905fbc01-a22f-4c3b-999b-ac3cbd69662e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:17.435875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:17.487136 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee5df47-2b6e-4431-a544-853fc1593a70 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:17.535564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2678f4-c381-4009-a510-a638df4c7aca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:17.536182 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2678f4-c381-4009-a510-a638df4c7aca None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:17.536495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2678f4-c381-4009-a510-a638df4c7aca None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:17.536769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2678f4-c381-4009-a510-a638df4c7aca None admin] PATH_INFO: `/v3/endpoints` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:17.537488 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2678f4-c381-4009-a510-a638df4c7aca None admin] RBAC: Adding query filter params (interface=public, service_id=e63dce484c74400ca8cee6d3abddc56c, region_id=RegionOne) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:17.537849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2678f4-c381-4009-a510-a638df4c7aca None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:17.539360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2678f4-c381-4009-a510-a638df4c7aca None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:17.549369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 389/771] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Sep 28 12:02:17 2022] GET /identity/v3/endpoints?service_id=e63dce484c74400ca8cee6d3abddc56c&interface=public®ion_id=RegionOne => generated 194 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:17.554166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2678f4-c381-4009-a510-a638df4c7aca None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:17.606216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905fbc01-a22f-4c3b-999b-ac3cbd69662e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:17.655531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb105e9-a1a5-4733-92c3-a9fd595b1cac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:17.656161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb105e9-a1a5-4733-92c3-a9fd595b1cac None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:17.656566 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb105e9-a1a5-4733-92c3-a9fd595b1cac None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:17.656842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb105e9-a1a5-4733-92c3-a9fd595b1cac None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:17.664436 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb105e9-a1a5-4733-92c3-a9fd595b1cac None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:17.666011 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb105e9-a1a5-4733-92c3-a9fd595b1cac None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-0eb105e9-a1a5-4733-92c3-a9fd595b1cac None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:17.679123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:17.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:17.680936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:17.680936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:17.680936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:17.680936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:17.680936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:17.680936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:17.680936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:17.680936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:17.680936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 383/772] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:17.681422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb105e9-a1a5-4733-92c3-a9fd595b1cac None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:17.681959 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2678f4-c381-4009-a510-a638df4c7aca None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:17.684287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:17.707570 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:17.719022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc16dd04-5b1f-48e6-8faa-ceaa6580c017 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:17.719741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc16dd04-5b1f-48e6-8faa-ceaa6580c017 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:17.720084 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc16dd04-5b1f-48e6-8faa-ceaa6580c017 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:17.720448 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc16dd04-5b1f-48e6-8faa-ceaa6580c017 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:17.721119 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc16dd04-5b1f-48e6-8faa-ceaa6580c017 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:17.723114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc16dd04-5b1f-48e6-8faa-ceaa6580c017 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:17.728689 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 390/773] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Sep 28 12:02:17 2022] GET /identity/v3/services => generated 772 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:17.729657 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc16dd04-5b1f-48e6-8faa-ceaa6580c017 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:17.804010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb105e9-a1a5-4733-92c3-a9fd595b1cac None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:17.840271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc16dd04-5b1f-48e6-8faa-ceaa6580c017 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:17.846034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfd7f63-5267-4677-8c02-1ff511daeae2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:17.846688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfd7f63-5267-4677-8c02-1ff511daeae2 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:17.846970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfd7f63-5267-4677-8c02-1ff511daeae2 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:17.847791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfd7f63-5267-4677-8c02-1ff511daeae2 None admin] PATH_INFO: `/v3/services/network` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:17.848506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfd7f63-5267-4677-8c02-1ff511daeae2 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:17.850139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfd7f63-5267-4677-8c02-1ff511daeae2 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-5dfd7f63-5267-4677-8c02-1ff511daeae2 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:17.860905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:17.863216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:17.865255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 384/774] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:17 2022] GET /identity/v3/services/network => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:17.867252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5788e475-cd52-4c96-bf3c-a6ca287463c2 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:17.867252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5788e475-cd52-4c96-bf3c-a6ca287463c2 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:17.867252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5788e475-cd52-4c96-bf3c-a6ca287463c2 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:17.867252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 385/775] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:17.870636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5788e475-cd52-4c96-bf3c-a6ca287463c2 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:17.893593 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e538190e-e9f0-47d1-bd25-887c33c6d3de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:17.894843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e538190e-e9f0-47d1-bd25-887c33c6d3de None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:17.895204 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e538190e-e9f0-47d1-bd25-887c33c6d3de None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:17.895489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e538190e-e9f0-47d1-bd25-887c33c6d3de None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:17.896209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e538190e-e9f0-47d1-bd25-887c33c6d3de None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:17.896696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e538190e-e9f0-47d1-bd25-887c33c6d3de None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:17.898717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e538190e-e9f0-47d1-bd25-887c33c6d3de None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:17.916022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 391/776] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:17.920453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d114c06c-f5ab-4b98-b0a0-bc339b83b0b8 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:17.920453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d114c06c-f5ab-4b98-b0a0-bc339b83b0b8 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:17.920453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d114c06c-f5ab-4b98-b0a0-bc339b83b0b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:17.937357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-d114c06c-f5ab-4b98-b0a0-bc339b83b0b8 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:17.938555 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d114c06c-f5ab-4b98-b0a0-bc339b83b0b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:17.971034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:17.990790 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:18.011999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 392/777] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:17 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:18.013007 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d114c06c-f5ab-4b98-b0a0-bc339b83b0b8 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:18.021338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5788e475-cd52-4c96-bf3c-a6ca287463c2 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:18.073055 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707b9af5-b7e0-4b3a-928b-9cf15b88b032 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:18.073726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707b9af5-b7e0-4b3a-928b-9cf15b88b032 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:18.074014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707b9af5-b7e0-4b3a-928b-9cf15b88b032 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:18.074291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707b9af5-b7e0-4b3a-928b-9cf15b88b032 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:18.074931 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707b9af5-b7e0-4b3a-928b-9cf15b88b032 None admin] RBAC: Adding query filter params (name=network) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:18.075235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707b9af5-b7e0-4b3a-928b-9cf15b88b032 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:18.076630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707b9af5-b7e0-4b3a-928b-9cf15b88b032 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:18.093278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 386/778] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Sep 28 12:02:17 2022] GET /identity/v3/services?name=network => generated 124 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:18.099370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707b9af5-b7e0-4b3a-928b-9cf15b88b032 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:18.143134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d114c06c-f5ab-4b98-b0a0-bc339b83b0b8 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:18.215153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129f31ff-9b53-4d25-92db-fa82169e204c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:18.217862 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129f31ff-9b53-4d25-92db-fa82169e204c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:18.218146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129f31ff-9b53-4d25-92db-fa82169e204c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:18.218417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129f31ff-9b53-4d25-92db-fa82169e204c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:18.219151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129f31ff-9b53-4d25-92db-fa82169e204c None admin] RBAC: Adding query filter params (role.id=ed4d262295434a72bb0a8b9a7ff12e71, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=b20a6fb4e7a546058f5fa314c9a59299) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:18.219462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129f31ff-9b53-4d25-92db-fa82169e204c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:18.225120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129f31ff-9b53-4d25-92db-fa82169e204c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:18.237167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 393/779] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:18 2022] GET /identity/v3/role_assignments?user.id=b20a6fb4e7a546058f5fa314c9a59299&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=ed4d262295434a72bb0a8b9a7ff12e71 => generated 259 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:18.273241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707b9af5-b7e0-4b3a-928b-9cf15b88b032 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:18.308210 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:18.315705 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129f31ff-9b53-4d25-92db-fa82169e204c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:18.316820 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b0cbf2-0240-4fe0-a2b3-e58696b3fb69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:18.317452 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b0cbf2-0240-4fe0-a2b3-e58696b3fb69 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:18.317728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b0cbf2-0240-4fe0-a2b3-e58696b3fb69 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:18.317995 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b0cbf2-0240-4fe0-a2b3-e58696b3fb69 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:18.318718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b0cbf2-0240-4fe0-a2b3-e58696b3fb69 None admin] RBAC: Adding query filter params (type=network) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:18.319020 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b0cbf2-0240-4fe0-a2b3-e58696b3fb69 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:18.320445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b0cbf2-0240-4fe0-a2b3-e58696b3fb69 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:18.327713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 387/780] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Sep 28 12:02:18 2022] GET /identity/v3/services?type=network => generated 350 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:18.338364 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b0cbf2-0240-4fe0-a2b3-e58696b3fb69 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:18.407990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:18.526401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b0cbf2-0240-4fe0-a2b3-e58696b3fb69 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:18.529931 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129f31ff-9b53-4d25-92db-fa82169e204c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:18.580496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aecc67a-282f-41d0-9335-061c15ed9f5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:18.581114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aecc67a-282f-41d0-9335-061c15ed9f5e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:18.581392 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aecc67a-282f-41d0-9335-061c15ed9f5e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:18.582219 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aecc67a-282f-41d0-9335-061c15ed9f5e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:18.593577 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aecc67a-282f-41d0-9335-061c15ed9f5e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-0aecc67a-282f-41d0-9335-061c15ed9f5e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:18.601704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:18.602688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:18.603538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:18.603538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:18.603538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:18.603538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:18.603538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:18.603538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:18.603538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:18.603538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:18.603538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:18.603538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:18.603538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:18.603538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:18.605561 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 394/781] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:02:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:18.611427 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aecc67a-282f-41d0-9335-061c15ed9f5e None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:18.616297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229799db-fd8c-4815-a722-8fc22aed6467 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:18.617120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229799db-fd8c-4815-a722-8fc22aed6467 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:18.617404 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229799db-fd8c-4815-a722-8fc22aed6467 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:18.617675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229799db-fd8c-4815-a722-8fc22aed6467 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:18.618394 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229799db-fd8c-4815-a722-8fc22aed6467 None admin] RBAC: Adding query filter params (interface=public, service_id=f4c5fc0228de4543af34b95961b05d5a, region_id=RegionOne) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:18.618713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229799db-fd8c-4815-a722-8fc22aed6467 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:18.627380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229799db-fd8c-4815-a722-8fc22aed6467 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:18.641945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 388/782] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Sep 28 12:02:18 2022] GET /identity/v3/endpoints?service_id=f4c5fc0228de4543af34b95961b05d5a&interface=public®ion_id=RegionOne => generated 194 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:18.648076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229799db-fd8c-4815-a722-8fc22aed6467 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:18.768578 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:18.813573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aecc67a-282f-41d0-9335-061c15ed9f5e None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:18.816593 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:18.856496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229799db-fd8c-4815-a722-8fc22aed6467 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:18.863951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92afee3e-806a-4078-ac0b-895e62c85c4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:18.864602 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92afee3e-806a-4078-ac0b-895e62c85c4d None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:18.864883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92afee3e-806a-4078-ac0b-895e62c85c4d None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:18.865157 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92afee3e-806a-4078-ac0b-895e62c85c4d None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:18.865829 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92afee3e-806a-4078-ac0b-895e62c85c4d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:18.866135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92afee3e-806a-4078-ac0b-895e62c85c4d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:18.867490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92afee3e-806a-4078-ac0b-895e62c85c4d None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:18.884030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 395/783] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:02:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:18.889960 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92afee3e-806a-4078-ac0b-895e62c85c4d None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:18.927337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a81ed6e-87c8-4101-b166-00d0fddc007d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:18.927953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a81ed6e-87c8-4101-b166-00d0fddc007d None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:18.928234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a81ed6e-87c8-4101-b166-00d0fddc007d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:18.928536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a81ed6e-87c8-4101-b166-00d0fddc007d None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:18.929150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a81ed6e-87c8-4101-b166-00d0fddc007d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:18.934768 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a81ed6e-87c8-4101-b166-00d0fddc007d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:18.944903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 389/784] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Sep 28 12:02:18 2022] GET /identity/v3/services => generated 772 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:19.038422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92afee3e-806a-4078-ac0b-895e62c85c4d None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:19.080480 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3089e062-2393-447a-8e5f-20ce79f6c4a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:19.081195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089e062-2393-447a-8e5f-20ce79f6c4a3 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:19.082088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089e062-2393-447a-8e5f-20ce79f6c4a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:19.082494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089e062-2393-447a-8e5f-20ce79f6c4a3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:19.094747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3089e062-2393-447a-8e5f-20ce79f6c4a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:19.096300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3089e062-2393-447a-8e5f-20ce79f6c4a3 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-3089e062-2393-447a-8e5f-20ce79f6c4a3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:19.114699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:19.116938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:19.118999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:19.118999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:19.118999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:19.118999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 396/785] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:19.125697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a81ed6e-87c8-4101-b166-00d0fddc007d None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:19.270330 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a81ed6e-87c8-4101-b166-00d0fddc007d None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:19.297209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:19.297537 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a4a1a5-14ee-4fd6-8671-0c9dd082ac14 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:19.298015 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a4a1a5-14ee-4fd6-8671-0c9dd082ac14 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:19.298375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a4a1a5-14ee-4fd6-8671-0c9dd082ac14 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:19.301743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 397/786] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:19 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:19.309594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3125d161-f8ea-4ae1-8558-7d3d6a530ae3 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:19.310441 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3125d161-f8ea-4ae1-8558-7d3d6a530ae3 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:19.310996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3125d161-f8ea-4ae1-8558-7d3d6a530ae3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:19.317618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a17248-fa1c-45d9-9ff1-70a2e304164d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:19.318228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a17248-fa1c-45d9-9ff1-70a2e304164d None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:19.318507 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a17248-fa1c-45d9-9ff1-70a2e304164d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:19.318781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a17248-fa1c-45d9-9ff1-70a2e304164d None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:19.319428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a17248-fa1c-45d9-9ff1-70a2e304164d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:19.319729 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a17248-fa1c-45d9-9ff1-70a2e304164d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:19.321132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a17248-fa1c-45d9-9ff1-70a2e304164d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:19.333874 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 390/787] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:19.334747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-3125d161-f8ea-4ae1-8558-7d3d6a530ae3 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:19.336907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-3125d161-f8ea-4ae1-8558-7d3d6a530ae3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:19.339657 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a17248-fa1c-45d9-9ff1-70a2e304164d None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:19.353302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:19.450478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 398/788] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:19 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:19.487165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a17248-fa1c-45d9-9ff1-70a2e304164d None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:19.533695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06207ab3-09db-419d-91e8-700e7e0e34ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:19.534325 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06207ab3-09db-419d-91e8-700e7e0e34ce None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:19.535235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06207ab3-09db-419d-91e8-700e7e0e34ce None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:19.535511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06207ab3-09db-419d-91e8-700e7e0e34ce None admin] PATH_INFO: `/v3/users/swift` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:19.547075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06207ab3-09db-419d-91e8-700e7e0e34ce None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:19.548606 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06207ab3-09db-419d-91e8-700e7e0e34ce None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-06207ab3-09db-419d-91e8-700e7e0e34ce None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:19.565705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:19.566791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:19.568334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:19.568334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:19.568334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:19.568334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:19.568334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:19.568334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:19.568334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:19.568334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:19.568334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 28 12:02:19.568334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:19.568334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 391/789] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:02:19 2022] GET /identity/v3/users/swift => generated 83 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:19.572580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3125d161-f8ea-4ae1-8558-7d3d6a530ae3 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:19.680269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3125d161-f8ea-4ae1-8558-7d3d6a530ae3 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:19.715961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e88e72c-0263-4d21-a7e3-21d83709c2b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:19.716624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e88e72c-0263-4d21-a7e3-21d83709c2b5 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:19.716911 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e88e72c-0263-4d21-a7e3-21d83709c2b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:19.717235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e88e72c-0263-4d21-a7e3-21d83709c2b5 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:19.717967 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e88e72c-0263-4d21-a7e3-21d83709c2b5 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:19.718273 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e88e72c-0263-4d21-a7e3-21d83709c2b5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:19.719677 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e88e72c-0263-4d21-a7e3-21d83709c2b5 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:19.738535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 399/790] 10.222.0.26 () {56 vars in 1135 bytes} [Wed Sep 28 12:02:19 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:19.743845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06207ab3-09db-419d-91e8-700e7e0e34ce None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:19.771094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e88e72c-0263-4d21-a7e3-21d83709c2b5 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:19.774906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:19.826168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:19.929112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e88e72c-0263-4d21-a7e3-21d83709c2b5 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:19.938847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06207ab3-09db-419d-91e8-700e7e0e34ce None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:19.982661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc9fbed-c503-456e-b6fb-6f354aef47cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:19.983791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc9fbed-c503-456e-b6fb-6f354aef47cf None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:19.984389 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc9fbed-c503-456e-b6fb-6f354aef47cf None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:19.984878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc9fbed-c503-456e-b6fb-6f354aef47cf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:20.001976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc9fbed-c503-456e-b6fb-6f354aef47cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:20.004488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc9fbed-c503-456e-b6fb-6f354aef47cf None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:20.019729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f767a4-7c65-47e9-9113-750df2148fc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:20.020386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f767a4-7c65-47e9-9113-750df2148fc1 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:20.020671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f767a4-7c65-47e9-9113-750df2148fc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:20.020954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f767a4-7c65-47e9-9113-750df2148fc1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-edc9fbed-c503-456e-b6fb-6f354aef47cf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:20.023942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:20.024933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:20.025993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:20.025993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:20.025993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:20.028056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 392/791] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:20.033292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc9fbed-c503-456e-b6fb-6f354aef47cf None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:20.049438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f767a4-7c65-47e9-9113-750df2148fc1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-a7f767a4-7c65-47e9-9113-750df2148fc1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:20.055141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:20.056629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:20.057538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:20.057538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:20.057538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:20.057538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:20.057538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:20.057538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:20.057538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:20.057538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:20.057538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:20.057538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:20.057538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:20.057538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:20.062333 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 400/792] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:02:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:20.067694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f767a4-7c65-47e9-9113-750df2148fc1 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:20.161084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:20.170910 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc9fbed-c503-456e-b6fb-6f354aef47cf None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:20.203649 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:20.215723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f34b9c-12a0-466e-993a-f1b48a596d54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:20.216754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f34b9c-12a0-466e-993a-f1b48a596d54 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:20.217306 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f34b9c-12a0-466e-993a-f1b48a596d54 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:20.217829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f34b9c-12a0-466e-993a-f1b48a596d54 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:20.218732 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f34b9c-12a0-466e-993a-f1b48a596d54 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:20.219256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f34b9c-12a0-466e-993a-f1b48a596d54 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:20.221166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f34b9c-12a0-466e-993a-f1b48a596d54 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:20.235191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 393/793] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:20.236920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd192d09-2e8a-4729-992c-0073426a93f8 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:20.237443 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd192d09-2e8a-4729-992c-0073426a93f8 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:20.238494 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd192d09-2e8a-4729-992c-0073426a93f8 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:20.240301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 394/794] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:20.241239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd192d09-2e8a-4729-992c-0073426a93f8 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:20.261749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f767a4-7c65-47e9-9113-750df2148fc1 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:20.307564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc48c13-6d90-4b0f-b939-1f89a51118ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:20.308181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc48c13-6d90-4b0f-b939-1f89a51118ab None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:20.308493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc48c13-6d90-4b0f-b939-1f89a51118ab None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:20.308770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc48c13-6d90-4b0f-b939-1f89a51118ab None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:20.309771 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc48c13-6d90-4b0f-b939-1f89a51118ab None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:20.310334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc48c13-6d90-4b0f-b939-1f89a51118ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:20.311697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc48c13-6d90-4b0f-b939-1f89a51118ab None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:20.324469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 401/795] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:02:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:20.326101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b85be7-2cc5-494d-b376-613beefc1a6e None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:20.326631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b85be7-2cc5-494d-b376-613beefc1a6e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:20.327128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b85be7-2cc5-494d-b376-613beefc1a6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:20.353994 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-b2b85be7-2cc5-494d-b376-613beefc1a6e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:20.356541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b2b85be7-2cc5-494d-b376-613beefc1a6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:20.366966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd192d09-2e8a-4729-992c-0073426a93f8 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:20.403920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec3a863-54ef-467e-a9b6-7345f0accb00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:20.404648 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec3a863-54ef-467e-a9b6-7345f0accb00 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:20.404927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec3a863-54ef-467e-a9b6-7345f0accb00 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:20.405198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec3a863-54ef-467e-a9b6-7345f0accb00 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:20.412798 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec3a863-54ef-467e-a9b6-7345f0accb00 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:20.414446 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec3a863-54ef-467e-a9b6-7345f0accb00 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-3ec3a863-54ef-467e-a9b6-7345f0accb00 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:20.426413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:20.427341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:20.428924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:20.428924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:20.428924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:20.428924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:20.428924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:20.428924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:20.428924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:20.428924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:20.428924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 395/796] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:20.428924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec3a863-54ef-467e-a9b6-7345f0accb00 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:20.443425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 402/797] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:20 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:20.444812 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b85be7-2cc5-494d-b376-613beefc1a6e None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:20.445322 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:20.511193 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:20.559667 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b85be7-2cc5-494d-b376-613beefc1a6e None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:20.564051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec3a863-54ef-467e-a9b6-7345f0accb00 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:20.596203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a917d5b-eba4-44c8-8550-f66b994f1bc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:20.597254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a917d5b-eba4-44c8-8550-f66b994f1bc6 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:20.597823 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a917d5b-eba4-44c8-8550-f66b994f1bc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:20.598359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a917d5b-eba4-44c8-8550-f66b994f1bc6 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:20.599290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a917d5b-eba4-44c8-8550-f66b994f1bc6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:20.599807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a917d5b-eba4-44c8-8550-f66b994f1bc6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:20.601494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a917d5b-eba4-44c8-8550-f66b994f1bc6 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:20.619401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b23a00e-b48a-4981-9778-1fe0038b2166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:20.619401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b23a00e-b48a-4981-9778-1fe0038b2166 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:20.619401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b23a00e-b48a-4981-9778-1fe0038b2166 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:20.619401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b23a00e-b48a-4981-9778-1fe0038b2166 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:20.620008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 403/798] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:20.621586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a917d5b-eba4-44c8-8550-f66b994f1bc6 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:20.629579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b23a00e-b48a-4981-9778-1fe0038b2166 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:20.631031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b23a00e-b48a-4981-9778-1fe0038b2166 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-4b23a00e-b48a-4981-9778-1fe0038b2166 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:20.660612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:20.661637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:20.662510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:20.662510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:20.662510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:20.668831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 396/799] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:20.674023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b23a00e-b48a-4981-9778-1fe0038b2166 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:20.742711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a917d5b-eba4-44c8-8550-f66b994f1bc6 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:20.760246 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:20.781259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6738794a-c834-40aa-95c9-799940ef6750 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:20.782393 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6738794a-c834-40aa-95c9-799940ef6750 None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:20.782881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6738794a-c834-40aa-95c9-799940ef6750 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:20.783361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6738794a-c834-40aa-95c9-799940ef6750 None admin] PATH_INFO: `/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/e663aad79bb8429aa223564a15332ee2/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:20.791852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6738794a-c834-40aa-95c9-799940ef6750 None admin] RBAC: Authorizing `identity:create_grant(project_id=8695d562e3e24c7c9300cf3a23cd7502, user_id=e663aad79bb8429aa223564a15332ee2, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:20.792655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:20.794173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6738794a-c834-40aa-95c9-799940ef6750 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:20.818280 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 404/800] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:02:20 2022] PUT /identity/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/e663aad79bb8429aa223564a15332ee2/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:02:20.820062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07746ab7-ab1b-4609-a5dd-a94fcd52bc30 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:20.820671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07746ab7-ab1b-4609-a5dd-a94fcd52bc30 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:20.820941 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07746ab7-ab1b-4609-a5dd-a94fcd52bc30 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:20.822901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 405/801] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:20.826522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b23a00e-b48a-4981-9778-1fe0038b2166 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:20.829618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da35515-d26a-4fd9-bbb2-5bac2896f7b6 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:20.829921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da35515-d26a-4fd9-bbb2-5bac2896f7b6 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:20.830189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da35515-d26a-4fd9-bbb2-5bac2896f7b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:20.855197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-7da35515-d26a-4fd9-bbb2-5bac2896f7b6 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:20.861125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7da35515-d26a-4fd9-bbb2-5bac2896f7b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:20.880006 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75fd5b9-5c10-4918-bb5a-be0a0e267049 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:20.880650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75fd5b9-5c10-4918-bb5a-be0a0e267049 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:20.880931 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75fd5b9-5c10-4918-bb5a-be0a0e267049 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:20.881306 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75fd5b9-5c10-4918-bb5a-be0a0e267049 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:20.882072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75fd5b9-5c10-4918-bb5a-be0a0e267049 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:20.882382 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75fd5b9-5c10-4918-bb5a-be0a0e267049 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:20.883790 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75fd5b9-5c10-4918-bb5a-be0a0e267049 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:20.900512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 397/802] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:20.901420 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75fd5b9-5c10-4918-bb5a-be0a0e267049 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:20.949849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 406/803] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:20 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:20.950656 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da35515-d26a-4fd9-bbb2-5bac2896f7b6 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:21.048282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75fd5b9-5c10-4918-bb5a-be0a0e267049 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:21.087884 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:21.098111 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d82563-dc15-4abc-ac66-3d797ae4c383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:21.099160 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d82563-dc15-4abc-ac66-3d797ae4c383 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:21.099726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d82563-dc15-4abc-ac66-3d797ae4c383 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:21.100010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d82563-dc15-4abc-ac66-3d797ae4c383 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:21.100695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d82563-dc15-4abc-ac66-3d797ae4c383 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:21.102854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d82563-dc15-4abc-ac66-3d797ae4c383 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-e0d82563-dc15-4abc-ac66-3d797ae4c383 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:21.115710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:21.116776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:21.117650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 398/804] 10.222.0.26 () {56 vars in 1129 bytes} [Wed Sep 28 12:02:20 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:21.124585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d82563-dc15-4abc-ac66-3d797ae4c383 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:21.126244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da35515-d26a-4fd9-bbb2-5bac2896f7b6 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:21.137225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:21.199968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec0d023-a369-461c-a967-ab8bd802bff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:21.200545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec0d023-a369-461c-a967-ab8bd802bff9 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:21.200826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec0d023-a369-461c-a967-ab8bd802bff9 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:21.201097 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec0d023-a369-461c-a967-ab8bd802bff9 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:21.212035 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec0d023-a369-461c-a967-ab8bd802bff9 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:21.213761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec0d023-a369-461c-a967-ab8bd802bff9 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-3ec0d023-a369-461c-a967-ab8bd802bff9 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:21.241152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:21.242166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:21.243099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:21.243099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:21.243099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:21.243099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:21.243099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:21.243099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:21.243099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:21.243099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:21.243099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:02:21.243099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:21.243099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 407/805] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Sep 28 12:02:20 2022] GET /identity/v3/users/nova => generated 82 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:21.249824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec0d023-a369-461c-a967-ab8bd802bff9 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:21.303158 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d82563-dc15-4abc-ac66-3d797ae4c383 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:21.340090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48d6bca-23c1-471c-a3a5-ef6db71940be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:21.341179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48d6bca-23c1-471c-a3a5-ef6db71940be None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:21.341672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48d6bca-23c1-471c-a3a5-ef6db71940be None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:21.342142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48d6bca-23c1-471c-a3a5-ef6db71940be None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:21.342991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48d6bca-23c1-471c-a3a5-ef6db71940be None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:21.343491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48d6bca-23c1-471c-a3a5-ef6db71940be None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:21.345120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48d6bca-23c1-471c-a3a5-ef6db71940be None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:21.350782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 399/806] 10.222.0.26 () {56 vars in 1124 bytes} [Wed Sep 28 12:02:21 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:21.351926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48d6bca-23c1-471c-a3a5-ef6db71940be None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:21.383959 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec0d023-a369-461c-a967-ab8bd802bff9 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:21.417851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:21.427369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a2cc75-16c1-47db-a012-d2368e82d0fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:21.428009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a2cc75-16c1-47db-a012-d2368e82d0fe None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:21.428291 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a2cc75-16c1-47db-a012-d2368e82d0fe None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:21.428588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a2cc75-16c1-47db-a012-d2368e82d0fe None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:21.429328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a2cc75-16c1-47db-a012-d2368e82d0fe None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:21.429633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a2cc75-16c1-47db-a012-d2368e82d0fe None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:21.431062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a2cc75-16c1-47db-a012-d2368e82d0fe None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:21.451882 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 408/807] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Sep 28 12:02:21 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:21.453353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a2cc75-16c1-47db-a012-d2368e82d0fe None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:21.476815 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48d6bca-23c1-471c-a3a5-ef6db71940be None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:21.484415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:21.537810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a096a3e-ebb4-4919-815e-4bd59c0ae723 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:21.538705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a096a3e-ebb4-4919-815e-4bd59c0ae723 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:21.538991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a096a3e-ebb4-4919-815e-4bd59c0ae723 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:21.539264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a096a3e-ebb4-4919-815e-4bd59c0ae723 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:21.553677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a096a3e-ebb4-4919-815e-4bd59c0ae723 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-4a096a3e-ebb4-4919-815e-4bd59c0ae723 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:21.560399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:21.561302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:21.562128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:21.562128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:21.562128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:21.562128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:21.562128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:21.562128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:21.562128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:21.562128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:21.562128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:21.562128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:21.562128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:21.562128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:21.562128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 400/808] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:02:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:21.562801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a096a3e-ebb4-4919-815e-4bd59c0ae723 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:21.577085 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a2cc75-16c1-47db-a012-d2368e82d0fe None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:21.619001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0027a5de-a17e-4183-8a6d-2f2148ee330a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:21.620051 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0027a5de-a17e-4183-8a6d-2f2148ee330a None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:21.620555 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0027a5de-a17e-4183-8a6d-2f2148ee330a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:21.621158 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0027a5de-a17e-4183-8a6d-2f2148ee330a None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:21.622021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0027a5de-a17e-4183-8a6d-2f2148ee330a None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:21.622522 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0027a5de-a17e-4183-8a6d-2f2148ee330a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:21.624748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0027a5de-a17e-4183-8a6d-2f2148ee330a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:21.631805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 409/809] 10.222.0.26 () {56 vars in 1124 bytes} [Wed Sep 28 12:02:21 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:21.632902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0027a5de-a17e-4183-8a6d-2f2148ee330a None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:21.693957 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a096a3e-ebb4-4919-815e-4bd59c0ae723 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:21.709201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:21.730898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae96ec3d-2b3a-4864-abed-8883d29ad82f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:21.731870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae96ec3d-2b3a-4864-abed-8883d29ad82f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:21.732398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae96ec3d-2b3a-4864-abed-8883d29ad82f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:21.732974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae96ec3d-2b3a-4864-abed-8883d29ad82f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:21.756215 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:21.774067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0027a5de-a17e-4183-8a6d-2f2148ee330a None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:21.812582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2951bce7-43c4-461d-acd0-bebbfe2b6333 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:21.813574 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2951bce7-43c4-461d-acd0-bebbfe2b6333 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:21.814116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2951bce7-43c4-461d-acd0-bebbfe2b6333 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:21.814590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2951bce7-43c4-461d-acd0-bebbfe2b6333 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:21.815448 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2951bce7-43c4-461d-acd0-bebbfe2b6333 None admin] RBAC: Adding query filter params (name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:21.816541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2951bce7-43c4-461d-acd0-bebbfe2b6333 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:21.816764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae96ec3d-2b3a-4864-abed-8883d29ad82f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:21.818242 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae96ec3d-2b3a-4864-abed-8883d29ad82f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:21.818609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2951bce7-43c4-461d-acd0-bebbfe2b6333 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:21.829267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 410/810] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:02:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:21.830746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2951bce7-43c4-461d-acd0-bebbfe2b6333 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:21.932741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 401/811] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:21 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:02:21.934045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae96ec3d-2b3a-4864-abed-8883d29ad82f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:21.946037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2951bce7-43c4-461d-acd0-bebbfe2b6333 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:21.953039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:21.981873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0230d53d-dd17-46bb-89f4-4afae58c24b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:21.982412 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0230d53d-dd17-46bb-89f4-4afae58c24b2 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:21.982688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0230d53d-dd17-46bb-89f4-4afae58c24b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:21.982961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0230d53d-dd17-46bb-89f4-4afae58c24b2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:21.983638 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0230d53d-dd17-46bb-89f4-4afae58c24b2 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:21.985049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0230d53d-dd17-46bb-89f4-4afae58c24b2 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:22.019751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 411/812] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:02:21 2022] POST /identity/v3/endpoints => generated 346 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:22.019751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c704f4c-6997-43aa-a7d9-c8d18aeb7831 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:22.019751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c704f4c-6997-43aa-a7d9-c8d18aeb7831 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:22.019751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c704f4c-6997-43aa-a7d9-c8d18aeb7831 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:22.019751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 412/813] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:22 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:22.019751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c704f4c-6997-43aa-a7d9-c8d18aeb7831 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:22.029507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:22.079622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae96ec3d-2b3a-4864-abed-8883d29ad82f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:22.136028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70cd766-6a69-4512-acc9-6655527a6929 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:22.137311 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70cd766-6a69-4512-acc9-6655527a6929 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:22.137642 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70cd766-6a69-4512-acc9-6655527a6929 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:22.137914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70cd766-6a69-4512-acc9-6655527a6929 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:22.152157 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70cd766-6a69-4512-acc9-6655527a6929 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:22.152417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c704f4c-6997-43aa-a7d9-c8d18aeb7831 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:22.153699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70cd766-6a69-4512-acc9-6655527a6929 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-a70cd766-6a69-4512-acc9-6655527a6929 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:22.178788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:22.179814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:22.180731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:22.180731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:22.180731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:22.180731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 402/814] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:22.184332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1145eb-7efd-43e2-a550-3026c1f4c3f0 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:22.184332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1145eb-7efd-43e2-a550-3026c1f4c3f0 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:22.184332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1145eb-7efd-43e2-a550-3026c1f4c3f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:22.199191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c635462e-4864-4734-b2f5-4052faf10199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:22.199888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c635462e-4864-4734-b2f5-4052faf10199 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:22.200203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c635462e-4864-4734-b2f5-4052faf10199 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:22.200543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c635462e-4864-4734-b2f5-4052faf10199 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:22.206959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-fb1145eb-7efd-43e2-a550-3026c1f4c3f0 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:22.209129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-fb1145eb-7efd-43e2-a550-3026c1f4c3f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:22.214903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c635462e-4864-4734-b2f5-4052faf10199 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:22.216548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c635462e-4864-4734-b2f5-4052faf10199 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-c635462e-4864-4734-b2f5-4052faf10199 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:22.234831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:22.235788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:22.236727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:22.236727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:22.236727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:22.238375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 413/815] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:22.242888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c635462e-4864-4734-b2f5-4052faf10199 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:22.285233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:22.298145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 403/816] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:22 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:22.299282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1145eb-7efd-43e2-a550-3026c1f4c3f0 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:22.327176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:22.384866 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c635462e-4864-4734-b2f5-4052faf10199 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:22.425490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9c356a-50a9-4eb7-a31b-c6589932ced3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:22.426169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9c356a-50a9-4eb7-a31b-c6589932ced3 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:22.426469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9c356a-50a9-4eb7-a31b-c6589932ced3 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:22.426772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9c356a-50a9-4eb7-a31b-c6589932ced3 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:22.430457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9c356a-50a9-4eb7-a31b-c6589932ced3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:22.430457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9c356a-50a9-4eb7-a31b-c6589932ced3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:22.430457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9c356a-50a9-4eb7-a31b-c6589932ced3 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:22.448352 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 414/817] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:22.451136 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9c356a-50a9-4eb7-a31b-c6589932ced3 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:22.502410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1145eb-7efd-43e2-a550-3026c1f4c3f0 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:22.579404 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9871078-825f-4488-aff3-d3ae0db51a86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:22.580045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9871078-825f-4488-aff3-d3ae0db51a86 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:22.580361 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9871078-825f-4488-aff3-d3ae0db51a86 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:22.580642 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9871078-825f-4488-aff3-d3ae0db51a86 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:22.581389 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9871078-825f-4488-aff3-d3ae0db51a86 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:22.582197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9871078-825f-4488-aff3-d3ae0db51a86 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:22.583577 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9871078-825f-4488-aff3-d3ae0db51a86 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:22.608128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 404/818] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:22.614080 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9871078-825f-4488-aff3-d3ae0db51a86 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:22.633159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9c356a-50a9-4eb7-a31b-c6589932ced3 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:22.666658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:22.673276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99deb586-5ebe-4370-a3b5-f3a64ebc4255 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:22.673986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99deb586-5ebe-4370-a3b5-f3a64ebc4255 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:22.674316 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99deb586-5ebe-4370-a3b5-f3a64ebc4255 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:22.674590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99deb586-5ebe-4370-a3b5-f3a64ebc4255 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:22.696577 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99deb586-5ebe-4370-a3b5-f3a64ebc4255 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:22.698440 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99deb586-5ebe-4370-a3b5-f3a64ebc4255 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:22.702487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-99deb586-5ebe-4370-a3b5-f3a64ebc4255 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:22.713711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:22.714690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:22.715640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:22.715640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:22.715640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:22.715640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:22.715640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:22.715640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:22.715640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:22.715640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:22.715640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 28 12:02:22.715640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:22.718470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 415/819] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Sep 28 12:02:22 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:22.719455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99deb586-5ebe-4370-a3b5-f3a64ebc4255 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:22.769137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9871078-825f-4488-aff3-d3ae0db51a86 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:22.808287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47b2006-c916-457e-84f6-c7a94016f7fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:22.809313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b2006-c916-457e-84f6-c7a94016f7fe None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:22.809851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b2006-c916-457e-84f6-c7a94016f7fe None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:22.810336 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b2006-c916-457e-84f6-c7a94016f7fe None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:22.818207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47b2006-c916-457e-84f6-c7a94016f7fe None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:22.819920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47b2006-c916-457e-84f6-c7a94016f7fe None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-b47b2006-c916-457e-84f6-c7a94016f7fe None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:22.831912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:22.833019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:22.833997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:22.833997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:22.833997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:22.833997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:22.833997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:22.833997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:22.833997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:22.833997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:22.840766 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 405/820] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:22.841121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99deb586-5ebe-4370-a3b5-f3a64ebc4255 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:22.842050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47b2006-c916-457e-84f6-c7a94016f7fe None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:22.888520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880eb905-710e-4d1c-ae3b-ce430ade3836 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:22.889604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880eb905-710e-4d1c-ae3b-ce430ade3836 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:22.890108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880eb905-710e-4d1c-ae3b-ce430ade3836 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:22.890664 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880eb905-710e-4d1c-ae3b-ce430ade3836 None admin] PATH_INFO: `/v3/services/network` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:22.891502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880eb905-710e-4d1c-ae3b-ce430ade3836 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:22.893093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880eb905-710e-4d1c-ae3b-ce430ade3836 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-880eb905-710e-4d1c-ae3b-ce430ade3836 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:22.902816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:22.903792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:22.905003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:22.905003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:22.905003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:22.905003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:22.905003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:22.905003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:22.905003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:22.905003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:22.905003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:22.905003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:22.905003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:22.905003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:22.905003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 28 12:02:22.905003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:22.912139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 416/821] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:22 2022] GET /identity/v3/services/network => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:22.913221 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880eb905-710e-4d1c-ae3b-ce430ade3836 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:22.962022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47b2006-c916-457e-84f6-c7a94016f7fe None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:22.964264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:22.977572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:23.001904 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5157da58-c98c-40d7-bcf5-a399a4f2e504 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:23.002892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5157da58-c98c-40d7-bcf5-a399a4f2e504 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:23.003922 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5157da58-c98c-40d7-bcf5-a399a4f2e504 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:23.004428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5157da58-c98c-40d7-bcf5-a399a4f2e504 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:23.005353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5157da58-c98c-40d7-bcf5-a399a4f2e504 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:23.005870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5157da58-c98c-40d7-bcf5-a399a4f2e504 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:23.007604 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5157da58-c98c-40d7-bcf5-a399a4f2e504 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:23.028213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 406/822] 10.222.0.26 () {56 vars in 1149 bytes} [Wed Sep 28 12:02:22 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:23.029567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5157da58-c98c-40d7-bcf5-a399a4f2e504 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:23.039505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880eb905-710e-4d1c-ae3b-ce430ade3836 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:23.086257 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5046e2b1-3087-47fc-852d-337d56d8c423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:23.086919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5046e2b1-3087-47fc-852d-337d56d8c423 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:23.087484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5046e2b1-3087-47fc-852d-337d56d8c423 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:23.087780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5046e2b1-3087-47fc-852d-337d56d8c423 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:23.088992 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5046e2b1-3087-47fc-852d-337d56d8c423 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:23.089318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5046e2b1-3087-47fc-852d-337d56d8c423 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:23.091110 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5046e2b1-3087-47fc-852d-337d56d8c423 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:23.112359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 417/823] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:23.113611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5046e2b1-3087-47fc-852d-337d56d8c423 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:23.148800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5157da58-c98c-40d7-bcf5-a399a4f2e504 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:23.189536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9804e860-d904-4e68-9ec0-5e1edfaa81ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:23.190921 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9804e860-d904-4e68-9ec0-5e1edfaa81ce None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:23.191521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9804e860-d904-4e68-9ec0-5e1edfaa81ce None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:23.191981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9804e860-d904-4e68-9ec0-5e1edfaa81ce None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:23.192977 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9804e860-d904-4e68-9ec0-5e1edfaa81ce None admin] RBAC: Adding query filter params (name=network) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:23.193758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9804e860-d904-4e68-9ec0-5e1edfaa81ce None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:23.195823 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9804e860-d904-4e68-9ec0-5e1edfaa81ce None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:23.202320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 407/824] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Sep 28 12:02:23 2022] GET /identity/v3/services?name=network => generated 124 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:23.203348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9804e860-d904-4e68-9ec0-5e1edfaa81ce None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:23.243854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:23.245337 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5046e2b1-3087-47fc-852d-337d56d8c423 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:23.246998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:23.282747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bf613a-d8d1-45d4-a73a-198b0d3af898 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:23.282747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bf613a-d8d1-45d4-a73a-198b0d3af898 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:23.283029 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bf613a-d8d1-45d4-a73a-198b0d3af898 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:23.283299 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bf613a-d8d1-45d4-a73a-198b0d3af898 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:23.294144 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bf613a-d8d1-45d4-a73a-198b0d3af898 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:23.295576 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bf613a-d8d1-45d4-a73a-198b0d3af898 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-57bf613a-d8d1-45d4-a73a-198b0d3af898 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:23.311719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:23.312764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:23.318764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:23.318764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:23.318764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:23.318764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 418/825] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:23.318998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9804e860-d904-4e68-9ec0-5e1edfaa81ce None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:23.319081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bf613a-d8d1-45d4-a73a-198b0d3af898 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:23.352472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896c2129-681f-4994-b355-0f6c1fcf484c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:23.353080 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896c2129-681f-4994-b355-0f6c1fcf484c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:23.353361 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896c2129-681f-4994-b355-0f6c1fcf484c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:23.353636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896c2129-681f-4994-b355-0f6c1fcf484c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:23.354362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896c2129-681f-4994-b355-0f6c1fcf484c None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=8bf969348e7a4158acf8fb789430b0e6) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:23.354668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896c2129-681f-4994-b355-0f6c1fcf484c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:23.356111 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896c2129-681f-4994-b355-0f6c1fcf484c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:23.362090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 408/826] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:23 2022] GET /identity/v3/role_assignments?user.id=8bf969348e7a4158acf8fb789430b0e6&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:23.363917 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7548a22-3934-4c2b-a2b9-ff949573fd9e None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:23.364205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7548a22-3934-4c2b-a2b9-ff949573fd9e None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:23.364604 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7548a22-3934-4c2b-a2b9-ff949573fd9e None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:23.368576 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 409/827] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:23.368576 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7548a22-3934-4c2b-a2b9-ff949573fd9e None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:23.458350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bf613a-d8d1-45d4-a73a-198b0d3af898 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:23.511491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:23.517730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7548a22-3934-4c2b-a2b9-ff949573fd9e None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:23.523281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff0f140-2082-4ca6-ab6c-0e3996f610cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:23.523970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff0f140-2082-4ca6-ab6c-0e3996f610cd None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:23.524253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff0f140-2082-4ca6-ab6c-0e3996f610cd None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:23.524664 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff0f140-2082-4ca6-ab6c-0e3996f610cd None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:23.525319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff0f140-2082-4ca6-ab6c-0e3996f610cd None admin] RBAC: Adding query filter params (type=network) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:23.525624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff0f140-2082-4ca6-ab6c-0e3996f610cd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:23.527077 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff0f140-2082-4ca6-ab6c-0e3996f610cd None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:23.535036 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 419/828] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Sep 28 12:02:23 2022] GET /identity/v3/services?type=network => generated 350 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:23.537165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9952b4e-0ab2-4941-b62a-ce39b768647a None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:23.537454 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9952b4e-0ab2-4941-b62a-ce39b768647a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:23.537730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9952b4e-0ab2-4941-b62a-ce39b768647a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:23.560484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-a9952b4e-0ab2-4941-b62a-ce39b768647a None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:23.562042 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a9952b4e-0ab2-4941-b62a-ce39b768647a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:23.567974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5882e9ec-92bc-4ee5-bd77-8c1c8366fcc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:23.570920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5882e9ec-92bc-4ee5-bd77-8c1c8366fcc7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:23.570920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5882e9ec-92bc-4ee5-bd77-8c1c8366fcc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:23.570920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5882e9ec-92bc-4ee5-bd77-8c1c8366fcc7 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:23.570920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5882e9ec-92bc-4ee5-bd77-8c1c8366fcc7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:23.570920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5882e9ec-92bc-4ee5-bd77-8c1c8366fcc7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:23.571445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5882e9ec-92bc-4ee5-bd77-8c1c8366fcc7 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:23.581078 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:23.585786 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 410/829] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:23.586838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5882e9ec-92bc-4ee5-bd77-8c1c8366fcc7 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:23.658335 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 420/830] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:23 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:23.659449 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9952b4e-0ab2-4941-b62a-ce39b768647a None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:23.731176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5882e9ec-92bc-4ee5-bd77-8c1c8366fcc7 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:23.776922 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e018aa29-bd60-4223-8217-fc486caf2e1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:23.777583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e018aa29-bd60-4223-8217-fc486caf2e1f None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:23.777873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e018aa29-bd60-4223-8217-fc486caf2e1f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:23.778146 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e018aa29-bd60-4223-8217-fc486caf2e1f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:23.778819 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e018aa29-bd60-4223-8217-fc486caf2e1f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:23.780204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e018aa29-bd60-4223-8217-fc486caf2e1f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:23.826282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 411/831] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:02:23 2022] POST /identity/v3/endpoints => generated 337 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:23.849280 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9952b4e-0ab2-4941-b62a-ce39b768647a None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:23.896438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a5917b-5b78-43e7-b1c2-ed632464bdf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:23.897522 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a5917b-5b78-43e7-b1c2-ed632464bdf8 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:23.898000 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a5917b-5b78-43e7-b1c2-ed632464bdf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:23.898502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a5917b-5b78-43e7-b1c2-ed632464bdf8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:23.907379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a5917b-5b78-43e7-b1c2-ed632464bdf8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:23.910359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a5917b-5b78-43e7-b1c2-ed632464bdf8 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-43a5917b-5b78-43e7-b1c2-ed632464bdf8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:23.924383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:23.925930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:23.926830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:23.926830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:23.926830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:23.926830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:23.926830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:23.926830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:23.926830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:23.926830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:23.929166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 421/832] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:23.937464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e018aa29-bd60-4223-8217-fc486caf2e1f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:23.965392 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:24.029614 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a5917b-5b78-43e7-b1c2-ed632464bdf8 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:24.034295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:24.083844 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e018aa29-bd60-4223-8217-fc486caf2e1f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:24.139964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f4b600-b0b2-42eb-8d79-12800da5a4d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:24.141146 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f4b600-b0b2-42eb-8d79-12800da5a4d8 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:24.141459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f4b600-b0b2-42eb-8d79-12800da5a4d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:24.141739 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f4b600-b0b2-42eb-8d79-12800da5a4d8 None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:24.142502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f4b600-b0b2-42eb-8d79-12800da5a4d8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:24.143228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f4b600-b0b2-42eb-8d79-12800da5a4d8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:24.145112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f4b600-b0b2-42eb-8d79-12800da5a4d8 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:24.163077 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a5917b-5b78-43e7-b1c2-ed632464bdf8 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:24.163431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 412/833] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:24.169749 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f4b600-b0b2-42eb-8d79-12800da5a4d8 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:24.204204 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79276b00-46af-4507-b891-3dd699cc8e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:24.205256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79276b00-46af-4507-b891-3dd699cc8e29 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:24.205755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79276b00-46af-4507-b891-3dd699cc8e29 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:24.206755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79276b00-46af-4507-b891-3dd699cc8e29 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:24.219529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79276b00-46af-4507-b891-3dd699cc8e29 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-79276b00-46af-4507-b891-3dd699cc8e29 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:24.227200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:24.228192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:24.229378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:24.229378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:24.229378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:24.229378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:24.229378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:24.229378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:24.229378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:24.229378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:24.229378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:24.229378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:24.229378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:24.229378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:24.231917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 422/834] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:02:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:24.242256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79276b00-46af-4507-b891-3dd699cc8e29 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:24.320747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:24.366615 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79276b00-46af-4507-b891-3dd699cc8e29 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:24.395615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:24.410260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3249482-2c49-4fe5-bf7e-5488e781b05f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:24.410886 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3249482-2c49-4fe5-bf7e-5488e781b05f None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:24.411166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3249482-2c49-4fe5-bf7e-5488e781b05f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:24.411440 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3249482-2c49-4fe5-bf7e-5488e781b05f None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:24.412234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3249482-2c49-4fe5-bf7e-5488e781b05f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:24.412583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3249482-2c49-4fe5-bf7e-5488e781b05f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:24.413946 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3249482-2c49-4fe5-bf7e-5488e781b05f None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:24.425500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f4b600-b0b2-42eb-8d79-12800da5a4d8 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:24.428391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 423/835] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:02:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:24.433487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3249482-2c49-4fe5-bf7e-5488e781b05f None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:24.463818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738e1508-f85f-4cb8-8ab5-0b04890b6c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:24.464497 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738e1508-f85f-4cb8-8ab5-0b04890b6c0d None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:24.464803 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738e1508-f85f-4cb8-8ab5-0b04890b6c0d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:24.465077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738e1508-f85f-4cb8-8ab5-0b04890b6c0d None admin] PATH_INFO: `/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/b20a6fb4e7a546058f5fa314c9a59299/roles/ed4d262295434a72bb0a8b9a7ff12e71` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:24.480757 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738e1508-f85f-4cb8-8ab5-0b04890b6c0d None admin] RBAC: Authorizing `identity:create_grant(project_id=8695d562e3e24c7c9300cf3a23cd7502, user_id=b20a6fb4e7a546058f5fa314c9a59299, role_id=ed4d262295434a72bb0a8b9a7ff12e71)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:24.482120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738e1508-f85f-4cb8-8ab5-0b04890b6c0d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:24.499233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 413/836] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:02:24 2022] PUT /identity/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/b20a6fb4e7a546058f5fa314c9a59299/roles/ed4d262295434a72bb0a8b9a7ff12e71 => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:02:24.500863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d2736d-391e-449c-8caf-4ffd299bf2da None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:24.501148 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d2736d-391e-449c-8caf-4ffd299bf2da None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:24.501423 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d2736d-391e-449c-8caf-4ffd299bf2da None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:24.504340 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 414/837] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:24.514224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81dd868-3b00-41b0-aae1-c20a6bea0ccd None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:24.514882 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81dd868-3b00-41b0-aae1-c20a6bea0ccd None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:24.514882 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81dd868-3b00-41b0-aae1-c20a6bea0ccd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:24.534192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-e81dd868-3b00-41b0-aae1-c20a6bea0ccd None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:24.536003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e81dd868-3b00-41b0-aae1-c20a6bea0ccd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:24.568129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3249482-2c49-4fe5-bf7e-5488e781b05f None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:24.598619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:24.609511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3745b283-7a92-4f91-ac57-3a1845d8f72b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:24.610155 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3745b283-7a92-4f91-ac57-3a1845d8f72b None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:24.610431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3745b283-7a92-4f91-ac57-3a1845d8f72b None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:24.610704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3745b283-7a92-4f91-ac57-3a1845d8f72b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:24.625998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3745b283-7a92-4f91-ac57-3a1845d8f72b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:24.627362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3745b283-7a92-4f91-ac57-3a1845d8f72b None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:24.636846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 415/838] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:24 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-3745b283-7a92-4f91-ac57-3a1845d8f72b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:24.649926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:24.650966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:24.651887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:24.651887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:24.651887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:24.655844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 424/839] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:24.657529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81dd868-3b00-41b0-aae1-c20a6bea0ccd None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:24.719748 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:24.768765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81dd868-3b00-41b0-aae1-c20a6bea0ccd None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:24.804581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ab0e09-53be-4c86-a149-b1aeec71b55a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:24.805187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ab0e09-53be-4c86-a149-b1aeec71b55a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:24.805467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ab0e09-53be-4c86-a149-b1aeec71b55a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:24.805739 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ab0e09-53be-4c86-a149-b1aeec71b55a None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:24.806382 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ab0e09-53be-4c86-a149-b1aeec71b55a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:24.806685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ab0e09-53be-4c86-a149-b1aeec71b55a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:24.808053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ab0e09-53be-4c86-a149-b1aeec71b55a None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:24.819829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 416/840] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:24.825424 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3745b283-7a92-4f91-ac57-3a1845d8f72b None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:24.929718 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3745b283-7a92-4f91-ac57-3a1845d8f72b None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:24.942612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ab0e09-53be-4c86-a149-b1aeec71b55a None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:24.965206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf3bb03-16a8-4f4f-843d-7ad6d5e7fd3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:24.966158 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf3bb03-16a8-4f4f-843d-7ad6d5e7fd3c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:24.966642 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf3bb03-16a8-4f4f-843d-7ad6d5e7fd3c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:24.967145 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf3bb03-16a8-4f4f-843d-7ad6d5e7fd3c None admin] PATH_INFO: `/v3/users/swift` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:24.976295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf3bb03-16a8-4f4f-843d-7ad6d5e7fd3c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:24.978662 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf3bb03-16a8-4f4f-843d-7ad6d5e7fd3c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-abf3bb03-16a8-4f4f-843d-7ad6d5e7fd3c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:24.990126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:24.991086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:24.992086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:24.992086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:24.992086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:24.992086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:24.992086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:24.992086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:24.992086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:24.992086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:24.992086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 28 12:02:24.992086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:24.993670 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 425/841] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:02:24 2022] GET /identity/v3/users/swift => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:25.000380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf3bb03-16a8-4f4f-843d-7ad6d5e7fd3c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:25.047958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:25.065752 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ab0e09-53be-4c86-a149-b1aeec71b55a None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:25.084665 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:25.110616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7a80ca-837a-4c56-bcd7-0c3bbe054a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:25.111758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7a80ca-837a-4c56-bcd7-0c3bbe054a69 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:25.112591 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7a80ca-837a-4c56-bcd7-0c3bbe054a69 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:25.113335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7a80ca-837a-4c56-bcd7-0c3bbe054a69 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:25.114279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7a80ca-837a-4c56-bcd7-0c3bbe054a69 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:25.116517 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7a80ca-837a-4c56-bcd7-0c3bbe054a69 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:25.120701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf3bb03-16a8-4f4f-843d-7ad6d5e7fd3c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-4a7a80ca-837a-4c56-bcd7-0c3bbe054a69 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:25.128281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:25.129493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:25.131064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:25.131064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:25.131064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:25.131064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:25.131064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:25.131064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:25.131064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:25.131064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:25.131064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:25.131064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:25.131064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:25.131064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:25.131064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 28 12:02:25.131064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:25.134240 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 417/842] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:02:24 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:25.140003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7a80ca-837a-4c56-bcd7-0c3bbe054a69 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:25.163126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de20324-de5e-486d-be98-24adbd691f66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:25.164028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de20324-de5e-486d-be98-24adbd691f66 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:25.164581 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de20324-de5e-486d-be98-24adbd691f66 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:25.165059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de20324-de5e-486d-be98-24adbd691f66 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:25.165968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de20324-de5e-486d-be98-24adbd691f66 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:25.166454 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de20324-de5e-486d-be98-24adbd691f66 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:25.168749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de20324-de5e-486d-be98-24adbd691f66 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:25.192884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 426/843] 10.222.0.26 () {56 vars in 1135 bytes} [Wed Sep 28 12:02:24 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:25.194587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de20324-de5e-486d-be98-24adbd691f66 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:25.253967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7a80ca-837a-4c56-bcd7-0c3bbe054a69 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:25.294843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3012af8d-4641-45ad-a3b3-2d80f1b6172f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:25.296213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3012af8d-4641-45ad-a3b3-2d80f1b6172f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:25.296936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3012af8d-4641-45ad-a3b3-2d80f1b6172f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:25.297383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3012af8d-4641-45ad-a3b3-2d80f1b6172f None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:25.298433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3012af8d-4641-45ad-a3b3-2d80f1b6172f None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:25.299370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3012af8d-4641-45ad-a3b3-2d80f1b6172f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:25.301573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3012af8d-4641-45ad-a3b3-2d80f1b6172f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:25.308570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 418/844] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:02:25 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:25.314089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3012af8d-4641-45ad-a3b3-2d80f1b6172f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:25.314283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:25.321993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de20324-de5e-486d-be98-24adbd691f66 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:25.362348 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c3ca29-6934-409e-9f27-4b58218a9294 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:25.363160 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:25.364129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c3ca29-6934-409e-9f27-4b58218a9294 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:25.364759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c3ca29-6934-409e-9f27-4b58218a9294 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:25.365231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c3ca29-6934-409e-9f27-4b58218a9294 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:25.443698 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3012af8d-4641-45ad-a3b3-2d80f1b6172f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:25.446351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c3ca29-6934-409e-9f27-4b58218a9294 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:25.448300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c3ca29-6934-409e-9f27-4b58218a9294 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:25.507553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0b013e-4f7f-4b39-a196-7d50780fea23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:25.509033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0b013e-4f7f-4b39-a196-7d50780fea23 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:25.509623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0b013e-4f7f-4b39-a196-7d50780fea23 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:25.510140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0b013e-4f7f-4b39-a196-7d50780fea23 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:25.511667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0b013e-4f7f-4b39-a196-7d50780fea23 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:25.512506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0b013e-4f7f-4b39-a196-7d50780fea23 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:25.514975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0b013e-4f7f-4b39-a196-7d50780fea23 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:25.523910 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 419/845] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:02:25 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:25.568343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:25.576285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 427/846] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:25 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:02:25.583266 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0b013e-4f7f-4b39-a196-7d50780fea23 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:25.701254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0b013e-4f7f-4b39-a196-7d50780fea23 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:25.710560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:25.742424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449cc01c-6ab0-469b-9832-1c7ed4ec1707 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:25.742424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449cc01c-6ab0-469b-9832-1c7ed4ec1707 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:25.742424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449cc01c-6ab0-469b-9832-1c7ed4ec1707 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:25.742780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449cc01c-6ab0-469b-9832-1c7ed4ec1707 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:25.754607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449cc01c-6ab0-469b-9832-1c7ed4ec1707 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:25.755837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449cc01c-6ab0-469b-9832-1c7ed4ec1707 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-449cc01c-6ab0-469b-9832-1c7ed4ec1707 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:25.772015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:25.773042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:25.773919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:25.773919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:25.773919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:25.773919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 420/847] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:25.775671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767c308f-31b6-40ed-800e-d1b7f80bff19 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:25.775956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767c308f-31b6-40ed-800e-d1b7f80bff19 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:25.776228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767c308f-31b6-40ed-800e-d1b7f80bff19 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:25.777688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 428/848] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:25.783115 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767c308f-31b6-40ed-800e-d1b7f80bff19 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:25.783570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f5dd23-94b9-42e7-92a7-f22cf76fa90e None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:25.784002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f5dd23-94b9-42e7-92a7-f22cf76fa90e None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:25.784479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f5dd23-94b9-42e7-92a7-f22cf76fa90e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:25.804381 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-77f5dd23-94b9-42e7-92a7-f22cf76fa90e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:25.805826 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-77f5dd23-94b9-42e7-92a7-f22cf76fa90e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:25.884421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 421/849] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:25 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:25.904096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767c308f-31b6-40ed-800e-d1b7f80bff19 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:25.940943 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f47e80-d0ae-457c-bb9c-fe7ece0a3e75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:25.941983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f47e80-d0ae-457c-bb9c-fe7ece0a3e75 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:25.942507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f47e80-d0ae-457c-bb9c-fe7ece0a3e75 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:25.942978 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f47e80-d0ae-457c-bb9c-fe7ece0a3e75 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:25.943884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f47e80-d0ae-457c-bb9c-fe7ece0a3e75 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:25.944432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f47e80-d0ae-457c-bb9c-fe7ece0a3e75 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:25.946101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f47e80-d0ae-457c-bb9c-fe7ece0a3e75 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:25.959173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 429/850] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:25.965156 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f5dd23-94b9-42e7-92a7-f22cf76fa90e None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:26.055121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:26.070512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f5dd23-94b9-42e7-92a7-f22cf76fa90e None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:26.106203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bc6d62-f35c-47a2-91dd-f76c3ed6b4d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:26.107188 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bc6d62-f35c-47a2-91dd-f76c3ed6b4d6 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:26.107665 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bc6d62-f35c-47a2-91dd-f76c3ed6b4d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:26.108133 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bc6d62-f35c-47a2-91dd-f76c3ed6b4d6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:26.115956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bc6d62-f35c-47a2-91dd-f76c3ed6b4d6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:26.117691 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bc6d62-f35c-47a2-91dd-f76c3ed6b4d6 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-10bc6d62-f35c-47a2-91dd-f76c3ed6b4d6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:26.129520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:26.130432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:26.131342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:26.131342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:26.131342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:26.131342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:26.131342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:26.131342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:26.131342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:26.131342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:26.133756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 422/851] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:26.140589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f47e80-d0ae-457c-bb9c-fe7ece0a3e75 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:26.167066 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bc6d62-f35c-47a2-91dd-f76c3ed6b4d6 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:26.200703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:26.251623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f47e80-d0ae-457c-bb9c-fe7ece0a3e75 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:26.283514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bc6d62-f35c-47a2-91dd-f76c3ed6b4d6 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:26.287801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802bf539-2953-411e-ac09-4d0fc366a767 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:26.288530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802bf539-2953-411e-ac09-4d0fc366a767 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:26.288933 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802bf539-2953-411e-ac09-4d0fc366a767 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:26.289270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802bf539-2953-411e-ac09-4d0fc366a767 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:26.290089 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802bf539-2953-411e-ac09-4d0fc366a767 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:26.290461 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802bf539-2953-411e-ac09-4d0fc366a767 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:26.291969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802bf539-2953-411e-ac09-4d0fc366a767 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:26.307118 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 430/852] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:26.313411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802bf539-2953-411e-ac09-4d0fc366a767 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:26.332297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28650bc4-0888-4ec9-9358-5c9681aa42be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:26.333223 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28650bc4-0888-4ec9-9358-5c9681aa42be None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:26.333501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28650bc4-0888-4ec9-9358-5c9681aa42be None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:26.333774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28650bc4-0888-4ec9-9358-5c9681aa42be None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:26.334606 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28650bc4-0888-4ec9-9358-5c9681aa42be None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:26.336414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28650bc4-0888-4ec9-9358-5c9681aa42be None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-28650bc4-0888-4ec9-9358-5c9681aa42be None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:26.351155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:26.352151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:26.353068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 423/853] 10.222.0.26 () {56 vars in 1132 bytes} [Wed Sep 28 12:02:26 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:26.359208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28650bc4-0888-4ec9-9358-5c9681aa42be None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:26.378134 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:26.434369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802bf539-2953-411e-ac09-4d0fc366a767 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:26.455075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:26.475153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28650bc4-0888-4ec9-9358-5c9681aa42be None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:26.478445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf8a112-a1f6-43c3-ada7-a87aa0a96b6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:26.479223 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf8a112-a1f6-43c3-ada7-a87aa0a96b6e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:26.479647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf8a112-a1f6-43c3-ada7-a87aa0a96b6e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:26.480065 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf8a112-a1f6-43c3-ada7-a87aa0a96b6e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:26.480992 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf8a112-a1f6-43c3-ada7-a87aa0a96b6e None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=e663aad79bb8429aa223564a15332ee2) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:26.481442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf8a112-a1f6-43c3-ada7-a87aa0a96b6e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:26.483028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf8a112-a1f6-43c3-ada7-a87aa0a96b6e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:26.490385 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 431/854] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:26 2022] GET /identity/v3/role_assignments?user.id=e663aad79bb8429aa223564a15332ee2&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:26.529170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1013c1a2-61e6-4eca-8ba3-4f758b83192c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:26.529866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1013c1a2-61e6-4eca-8ba3-4f758b83192c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:26.530123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1013c1a2-61e6-4eca-8ba3-4f758b83192c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:26.530443 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1013c1a2-61e6-4eca-8ba3-4f758b83192c None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:26.531092 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1013c1a2-61e6-4eca-8ba3-4f758b83192c None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:26.531389 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1013c1a2-61e6-4eca-8ba3-4f758b83192c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:26.532920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1013c1a2-61e6-4eca-8ba3-4f758b83192c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:26.543327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 424/855] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Sep 28 12:02:26 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:26.544841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf8a112-a1f6-43c3-ada7-a87aa0a96b6e None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:26.657767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf8a112-a1f6-43c3-ada7-a87aa0a96b6e None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:26.695157 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa89a56-0f09-4d7d-bafd-ff87311e5b25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:26.695762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa89a56-0f09-4d7d-bafd-ff87311e5b25 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:26.696040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa89a56-0f09-4d7d-bafd-ff87311e5b25 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:26.696327 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa89a56-0f09-4d7d-bafd-ff87311e5b25 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:26.697040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa89a56-0f09-4d7d-bafd-ff87311e5b25 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:26.697343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa89a56-0f09-4d7d-bafd-ff87311e5b25 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:26.698695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa89a56-0f09-4d7d-bafd-ff87311e5b25 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:26.708394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 432/856] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Sep 28 12:02:26 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:26.846719 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2da1277-6b6b-4410-8a1e-ab96e751ba46 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:26.847602 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2da1277-6b6b-4410-8a1e-ab96e751ba46 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:26.847810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2da1277-6b6b-4410-8a1e-ab96e751ba46 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:26.849418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 425/857] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:26.855170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf4ca01-467b-4b09-a541-d25e0bd25c3d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:26.855663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf4ca01-467b-4b09-a541-d25e0bd25c3d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:26.856091 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf4ca01-467b-4b09-a541-d25e0bd25c3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:26.876834 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-2cf4ca01-467b-4b09-a541-d25e0bd25c3d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:26.878350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2cf4ca01-467b-4b09-a541-d25e0bd25c3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:26.926779 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:26.955822 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 433/858] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:26 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:27.225153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2da1277-6b6b-4410-8a1e-ab96e751ba46 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:27.308059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:27.332635 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2da1277-6b6b-4410-8a1e-ab96e751ba46 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:27.370968 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df9f72d-41f9-4ef6-bb0a-93c2281faa18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:27.371586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df9f72d-41f9-4ef6-bb0a-93c2281faa18 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:27.371897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df9f72d-41f9-4ef6-bb0a-93c2281faa18 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:27.372127 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df9f72d-41f9-4ef6-bb0a-93c2281faa18 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:27.382951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df9f72d-41f9-4ef6-bb0a-93c2281faa18 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-4df9f72d-41f9-4ef6-bb0a-93c2281faa18 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:27.387612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:27.388565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:27.389418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:27.389418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:27.389418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:27.389418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:27.389418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:27.389418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:27.389418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:27.389418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:27.389418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:27.389418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:27.389418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:02:27.389418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:27.389418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 426/859] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:02:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:27.396429 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf4ca01-467b-4b09-a541-d25e0bd25c3d None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:27.506321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf4ca01-467b-4b09-a541-d25e0bd25c3d None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:27.543069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b9b32b-0cc1-43e0-a51d-40981ec69ad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:27.544020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b9b32b-0cc1-43e0-a51d-40981ec69ad0 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:27.544542 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b9b32b-0cc1-43e0-a51d-40981ec69ad0 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:27.545050 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b9b32b-0cc1-43e0-a51d-40981ec69ad0 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:27.545918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b9b32b-0cc1-43e0-a51d-40981ec69ad0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:27.546440 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b9b32b-0cc1-43e0-a51d-40981ec69ad0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:27.548003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b9b32b-0cc1-43e0-a51d-40981ec69ad0 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:27.557439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 434/860] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:02:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:27.563417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df9f72d-41f9-4ef6-bb0a-93c2281faa18 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:27.669060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df9f72d-41f9-4ef6-bb0a-93c2281faa18 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:27.709437 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1917f9-b32b-4e48-964b-7b58708623bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:27.710521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1917f9-b32b-4e48-964b-7b58708623bd None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:27.711007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1917f9-b32b-4e48-964b-7b58708623bd None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:27.711496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1917f9-b32b-4e48-964b-7b58708623bd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:27.714801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:27.729430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1917f9-b32b-4e48-964b-7b58708623bd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:27.730918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1917f9-b32b-4e48-964b-7b58708623bd None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-dd1917f9-b32b-4e48-964b-7b58708623bd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:27.748045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:27.749070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:27.749984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:27.749984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:27.749984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:27.749984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 427/861] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:27.756579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b9b32b-0cc1-43e0-a51d-40981ec69ad0 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:27.794385 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:27.847471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004db08d-0e05-4bfa-bdfd-836e52a712de None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:27.847764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004db08d-0e05-4bfa-bdfd-836e52a712de None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:27.848038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004db08d-0e05-4bfa-bdfd-836e52a712de None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:27.849687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 428/862] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:27.855659 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d04a08-7a85-40f2-8ea5-04ec63ce0704 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:27.855947 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d04a08-7a85-40f2-8ea5-04ec63ce0704 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:27.856219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d04a08-7a85-40f2-8ea5-04ec63ce0704 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:27.870356 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b9b32b-0cc1-43e0-a51d-40981ec69ad0 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:27.875627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-37d04a08-7a85-40f2-8ea5-04ec63ce0704 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:27.876851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-37d04a08-7a85-40f2-8ea5-04ec63ce0704 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:27.906189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8fdd49-723c-4b88-92c0-f46c0928cfd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:27.907158 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8fdd49-723c-4b88-92c0-f46c0928cfd7 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:27.907621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8fdd49-723c-4b88-92c0-f46c0928cfd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:27.908057 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8fdd49-723c-4b88-92c0-f46c0928cfd7 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:27.908901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8fdd49-723c-4b88-92c0-f46c0928cfd7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:27.909419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8fdd49-723c-4b88-92c0-f46c0928cfd7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:27.911407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8fdd49-723c-4b88-92c0-f46c0928cfd7 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:27.925170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 435/863] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:27.930689 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8fdd49-723c-4b88-92c0-f46c0928cfd7 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:27.964557 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 429/864] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:27 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:28.029966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:28.037031 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8fdd49-723c-4b88-92c0-f46c0928cfd7 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:28.071999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703c7623-23eb-4c55-ae39-fe712159ded8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:28.072658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703c7623-23eb-4c55-ae39-fe712159ded8 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:28.072936 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703c7623-23eb-4c55-ae39-fe712159ded8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:28.073207 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703c7623-23eb-4c55-ae39-fe712159ded8 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:28.080348 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703c7623-23eb-4c55-ae39-fe712159ded8 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:28.082275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703c7623-23eb-4c55-ae39-fe712159ded8 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-703c7623-23eb-4c55-ae39-fe712159ded8 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:28.094239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:28.095176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:28.096112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:28.096112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:28.096112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:28.096112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:28.096112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:28.096112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:28.096112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:28.096112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:28.096112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 28 12:02:28.096112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:28.096112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 436/865] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Sep 28 12:02:27 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:28.101940 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d04a08-7a85-40f2-8ea5-04ec63ce0704 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:28.203611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d04a08-7a85-40f2-8ea5-04ec63ce0704 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:28.225822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:28.242938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8155353a-9343-4097-94c3-483826c95c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:28.243537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8155353a-9343-4097-94c3-483826c95c0d None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:28.243814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8155353a-9343-4097-94c3-483826c95c0d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:28.244083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8155353a-9343-4097-94c3-483826c95c0d None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:28.244840 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8155353a-9343-4097-94c3-483826c95c0d None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:28.245139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8155353a-9343-4097-94c3-483826c95c0d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:28.246561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8155353a-9343-4097-94c3-483826c95c0d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:28.249076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703c7623-23eb-4c55-ae39-fe712159ded8 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:28.264821 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 430/866] 10.222.0.26 () {56 vars in 1149 bytes} [Wed Sep 28 12:02:28 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:28.270897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8155353a-9343-4097-94c3-483826c95c0d None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:28.369841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703c7623-23eb-4c55-ae39-fe712159ded8 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:28.399015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8155353a-9343-4097-94c3-483826c95c0d None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:28.409051 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f088e02f-5f12-4db0-a4c9-f9114fdafe95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:28.410208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f088e02f-5f12-4db0-a4c9-f9114fdafe95 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:28.410484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f088e02f-5f12-4db0-a4c9-f9114fdafe95 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:28.410753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f088e02f-5f12-4db0-a4c9-f9114fdafe95 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:28.411768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f088e02f-5f12-4db0-a4c9-f9114fdafe95 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:28.413550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f088e02f-5f12-4db0-a4c9-f9114fdafe95 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:28.433300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 437/867] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Sep 28 12:02:28 2022] POST /identity/v3/services => generated 251 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:28.451104 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e12bde-b1b6-4fcc-ae62-9c5e5f3a97e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:28.452067 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e12bde-b1b6-4fcc-ae62-9c5e5f3a97e8 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:28.452601 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e12bde-b1b6-4fcc-ae62-9c5e5f3a97e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:28.453121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e12bde-b1b6-4fcc-ae62-9c5e5f3a97e8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:28.508290 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:28.540472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e12bde-b1b6-4fcc-ae62-9c5e5f3a97e8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:28.542017 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e12bde-b1b6-4fcc-ae62-9c5e5f3a97e8 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:28.652768 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 431/868] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:28 2022] GET /identity/v3/auth/tokens => generated 1262 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 28 12:02:28.658650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f088e02f-5f12-4db0-a4c9-f9114fdafe95 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:28.678271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:28.769651 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f088e02f-5f12-4db0-a4c9-f9114fdafe95 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:28.803988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317a8d77-36e0-4481-a55e-07ca34950e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:28.804641 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317a8d77-36e0-4481-a55e-07ca34950e39 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:28.804917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317a8d77-36e0-4481-a55e-07ca34950e39 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:28.805186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317a8d77-36e0-4481-a55e-07ca34950e39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:28.815911 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317a8d77-36e0-4481-a55e-07ca34950e39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:28.817372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317a8d77-36e0-4481-a55e-07ca34950e39 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-317a8d77-36e0-4481-a55e-07ca34950e39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:28.832452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:28.833455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:28.834331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:28.834331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:28.834331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:28.834331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 438/869] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:28.841222 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e12bde-b1b6-4fcc-ae62-9c5e5f3a97e8 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:28.854578 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4948ea8f-aa15-43d9-88b1-ee915297d235 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:28.854859 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4948ea8f-aa15-43d9-88b1-ee915297d235 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:28.855061 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4948ea8f-aa15-43d9-88b1-ee915297d235 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:28.856668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 439/870] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:28.861675 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c96ae9-fe55-4d6f-8e9d-2f02b6929b29 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:28.861963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c96ae9-fe55-4d6f-8e9d-2f02b6929b29 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:28.862239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c96ae9-fe55-4d6f-8e9d-2f02b6929b29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:28.881476 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-54c96ae9-fe55-4d6f-8e9d-2f02b6929b29 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:28.882055 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-54c96ae9-fe55-4d6f-8e9d-2f02b6929b29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:28.925983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:28.961229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e12bde-b1b6-4fcc-ae62-9c5e5f3a97e8 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:28.963791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:28.976394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 440/871] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:28 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:29.005393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f800e994-0f4b-4195-b167-e6c544d2c3fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:29.006380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f800e994-0f4b-4195-b167-e6c544d2c3fa None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:29.006930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f800e994-0f4b-4195-b167-e6c544d2c3fa None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:29.007409 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f800e994-0f4b-4195-b167-e6c544d2c3fa None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:29.008348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f800e994-0f4b-4195-b167-e6c544d2c3fa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:29.008859 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f800e994-0f4b-4195-b167-e6c544d2c3fa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:29.010564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f800e994-0f4b-4195-b167-e6c544d2c3fa None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:29.019237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1217502f-8a76-4ca2-ba73-8c7ae7a1ad70 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:29.019842 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1217502f-8a76-4ca2-ba73-8c7ae7a1ad70 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:29.020349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1217502f-8a76-4ca2-ba73-8c7ae7a1ad70 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:29.022189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 441/872] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:29.025238 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 432/873] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:29.030715 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1217502f-8a76-4ca2-ba73-8c7ae7a1ad70 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:29.031459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4bc30f-ad9b-491b-89a1-fc2c81b23cf6 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:29.031953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4bc30f-ad9b-491b-89a1-fc2c81b23cf6 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:29.032465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4bc30f-ad9b-491b-89a1-fc2c81b23cf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:29.055585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-9e4bc30f-ad9b-491b-89a1-fc2c81b23cf6 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:29.058023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9e4bc30f-ad9b-491b-89a1-fc2c81b23cf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:29.144735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 433/874] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:29 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:29.157146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1217502f-8a76-4ca2-ba73-8c7ae7a1ad70 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:29.193244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7ad1b2-9770-45bf-9314-90251e72c965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:29.194232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7ad1b2-9770-45bf-9314-90251e72c965 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:29.194719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7ad1b2-9770-45bf-9314-90251e72c965 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:29.195200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7ad1b2-9770-45bf-9314-90251e72c965 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:29.202187 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7ad1b2-9770-45bf-9314-90251e72c965 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:29.203837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7ad1b2-9770-45bf-9314-90251e72c965 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-3d7ad1b2-9770-45bf-9314-90251e72c965 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:29.216838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:29.217717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:29.218606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:29.218606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:29.218606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:29.218606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:29.218606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:29.218606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:29.218606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:29.218606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:29.221008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 442/875] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:29.227422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4bc30f-ad9b-491b-89a1-fc2c81b23cf6 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:29.262798 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7ad1b2-9770-45bf-9314-90251e72c965 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:29.304042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:29.327665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:29.359428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4bc30f-ad9b-491b-89a1-fc2c81b23cf6 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:29.377755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7ad1b2-9770-45bf-9314-90251e72c965 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:29.398789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd0a4eb-c32b-46a3-8630-a5e98beee8a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:29.399803 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd0a4eb-c32b-46a3-8630-a5e98beee8a4 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:29.400289 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd0a4eb-c32b-46a3-8630-a5e98beee8a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:29.400807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd0a4eb-c32b-46a3-8630-a5e98beee8a4 None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:29.401721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd0a4eb-c32b-46a3-8630-a5e98beee8a4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:29.402230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd0a4eb-c32b-46a3-8630-a5e98beee8a4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:29.404066 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd0a4eb-c32b-46a3-8630-a5e98beee8a4 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:29.414810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dd1c21-8cd9-4196-84d2-58af032790ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:29.415450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dd1c21-8cd9-4196-84d2-58af032790ab None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:29.415732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dd1c21-8cd9-4196-84d2-58af032790ab None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:29.416008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dd1c21-8cd9-4196-84d2-58af032790ab None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:29.416629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dd1c21-8cd9-4196-84d2-58af032790ab None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:29.417969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 434/876] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:29.418161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dd1c21-8cd9-4196-84d2-58af032790ab None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:29.419204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd0a4eb-c32b-46a3-8630-a5e98beee8a4 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-25dd1c21-8cd9-4196-84d2-58af032790ab None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:29.428939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:29.430043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:29.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:29.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:29.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:29.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:29.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:29.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:29.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:29.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:29.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:29.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:29.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:29.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:29.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:29.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:29.435532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 443/877] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Sep 28 12:02:29 2022] GET /identity/v3/services/object-store => generated 93 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:29.436362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dd1c21-8cd9-4196-84d2-58af032790ab None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:29.526586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd0a4eb-c32b-46a3-8630-a5e98beee8a4 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:29.550217 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:29.550350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dd1c21-8cd9-4196-84d2-58af032790ab None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:29.564162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a185f93-8fb0-41fa-82e0-75ea44ba223d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:29.564796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a185f93-8fb0-41fa-82e0-75ea44ba223d None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:29.565075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a185f93-8fb0-41fa-82e0-75ea44ba223d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:29.565347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a185f93-8fb0-41fa-82e0-75ea44ba223d None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:29.566020 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a185f93-8fb0-41fa-82e0-75ea44ba223d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:29.567376 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a185f93-8fb0-41fa-82e0-75ea44ba223d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:29.573550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:29.584586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 435/878] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Sep 28 12:02:29 2022] POST /identity/v3/services => generated 269 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:29.585674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a185f93-8fb0-41fa-82e0-75ea44ba223d None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:29.607018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3101d460-d985-4d75-a15c-0cf23f48aa3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:29.608069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3101d460-d985-4d75-a15c-0cf23f48aa3c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:29.608428 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3101d460-d985-4d75-a15c-0cf23f48aa3c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:29.608772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3101d460-d985-4d75-a15c-0cf23f48aa3c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:29.609703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3101d460-d985-4d75-a15c-0cf23f48aa3c None admin] RBAC: Adding query filter params (role.id=ed4d262295434a72bb0a8b9a7ff12e71, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=b20a6fb4e7a546058f5fa314c9a59299) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:29.610200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3101d460-d985-4d75-a15c-0cf23f48aa3c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:29.611872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3101d460-d985-4d75-a15c-0cf23f48aa3c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:29.628707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 444/879] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:29 2022] GET /identity/v3/role_assignments?user.id=b20a6fb4e7a546058f5fa314c9a59299&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=ed4d262295434a72bb0a8b9a7ff12e71 => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:29.735116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a185f93-8fb0-41fa-82e0-75ea44ba223d None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:29.778490 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575195b3-240d-4d66-8ddd-57c6ef6cc41c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:29.779126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575195b3-240d-4d66-8ddd-57c6ef6cc41c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:29.779408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575195b3-240d-4d66-8ddd-57c6ef6cc41c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:29.779715 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575195b3-240d-4d66-8ddd-57c6ef6cc41c None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:29.780383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575195b3-240d-4d66-8ddd-57c6ef6cc41c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:29.780697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575195b3-240d-4d66-8ddd-57c6ef6cc41c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:29.782197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575195b3-240d-4d66-8ddd-57c6ef6cc41c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:29.788450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 436/880] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Sep 28 12:02:29 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:29.794260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3101d460-d985-4d75-a15c-0cf23f48aa3c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:29.902977 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3101d460-d985-4d75-a15c-0cf23f48aa3c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:29.937419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00fb8d2-fb9d-4fcd-ab5d-3454f27847fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:29.938035 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00fb8d2-fb9d-4fcd-ab5d-3454f27847fe None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:29.938309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00fb8d2-fb9d-4fcd-ab5d-3454f27847fe None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:29.938644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00fb8d2-fb9d-4fcd-ab5d-3454f27847fe None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:29.939282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00fb8d2-fb9d-4fcd-ab5d-3454f27847fe None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:29.939589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00fb8d2-fb9d-4fcd-ab5d-3454f27847fe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:29.941543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00fb8d2-fb9d-4fcd-ab5d-3454f27847fe None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:29.946434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 445/881] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Sep 28 12:02:29 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:30.730172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd48ec16-6e6c-480c-998f-9c851963d5f1 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:30.730549 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd48ec16-6e6c-480c-998f-9c851963d5f1 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:30.730729 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd48ec16-6e6c-480c-998f-9c851963d5f1 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:30.732299 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 437/882] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:30.738160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866d651b-07f4-41cf-a694-322b6ff3a1aa None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:30.738447 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866d651b-07f4-41cf-a694-322b6ff3a1aa None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:30.738720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866d651b-07f4-41cf-a694-322b6ff3a1aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:30.756798 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-866d651b-07f4-41cf-a694-322b6ff3a1aa None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:30.757895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-866d651b-07f4-41cf-a694-322b6ff3a1aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:30.801089 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:30.826628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 446/883] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:30 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:31.104054 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd48ec16-6e6c-480c-998f-9c851963d5f1 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:31.154170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:31.215256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd48ec16-6e6c-480c-998f-9c851963d5f1 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:31.250357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f035572c-c5af-4c0f-9970-923f4de54a21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:31.251349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f035572c-c5af-4c0f-9970-923f4de54a21 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:31.251846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f035572c-c5af-4c0f-9970-923f4de54a21 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:31.252356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f035572c-c5af-4c0f-9970-923f4de54a21 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:31.253722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f035572c-c5af-4c0f-9970-923f4de54a21 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:31.255320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f035572c-c5af-4c0f-9970-923f4de54a21 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-f035572c-c5af-4c0f-9970-923f4de54a21 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:31.264400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:31.265435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:31.266328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:31.266328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:31.266328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:31.266328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:31.266328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:31.266328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:31.266328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:31.266328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:31.266328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:31.266328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:31.266328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:31.266328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:31.266328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 28 12:02:31.266328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:31.269669 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 438/884] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:02:31 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:31.277180 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866d651b-07f4-41cf-a694-322b6ff3a1aa None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:31.373737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866d651b-07f4-41cf-a694-322b6ff3a1aa None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:31.409707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a090ff97-aed4-46c4-b857-b7ee8b8fee23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:31.410335 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a090ff97-aed4-46c4-b857-b7ee8b8fee23 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:31.410615 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a090ff97-aed4-46c4-b857-b7ee8b8fee23 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:31.410899 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a090ff97-aed4-46c4-b857-b7ee8b8fee23 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:31.411515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a090ff97-aed4-46c4-b857-b7ee8b8fee23 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:31.411804 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a090ff97-aed4-46c4-b857-b7ee8b8fee23 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:31.413295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a090ff97-aed4-46c4-b857-b7ee8b8fee23 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:31.418616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 447/885] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:02:31 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:31.424222 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f035572c-c5af-4c0f-9970-923f4de54a21 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:31.533499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f035572c-c5af-4c0f-9970-923f4de54a21 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:31.544866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:31.569890 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c91cb1-6b32-488f-8405-74285aa1078b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:31.570542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c91cb1-6b32-488f-8405-74285aa1078b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:31.570849 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c91cb1-6b32-488f-8405-74285aa1078b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:31.571090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c91cb1-6b32-488f-8405-74285aa1078b None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:31.571719 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c91cb1-6b32-488f-8405-74285aa1078b None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:31.572016 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c91cb1-6b32-488f-8405-74285aa1078b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:31.573481 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c91cb1-6b32-488f-8405-74285aa1078b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:31.578718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 439/886] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:02:31 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:31.584345 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a090ff97-aed4-46c4-b857-b7ee8b8fee23 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:31.630227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:31.694832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a090ff97-aed4-46c4-b857-b7ee8b8fee23 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:31.732716 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d09ceb8-55a0-435b-bc90-892eeef80b08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:31.733648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d09ceb8-55a0-435b-bc90-892eeef80b08 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:31.734078 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d09ceb8-55a0-435b-bc90-892eeef80b08 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:31.734475 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d09ceb8-55a0-435b-bc90-892eeef80b08 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:31.735526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d09ceb8-55a0-435b-bc90-892eeef80b08 None admin] RBAC: Adding query filter params (interface=public, service_id=1e99a148dc944efc99d02e757975d648, region_id=RegionOne) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:31.735995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d09ceb8-55a0-435b-bc90-892eeef80b08 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:31.738182 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d09ceb8-55a0-435b-bc90-892eeef80b08 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:31.746291 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 448/887] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Sep 28 12:02:31 2022] GET /identity/v3/endpoints?service_id=1e99a148dc944efc99d02e757975d648&interface=public®ion_id=RegionOne => generated 194 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:31.754077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c91cb1-6b32-488f-8405-74285aa1078b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:31.869641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c91cb1-6b32-488f-8405-74285aa1078b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:31.892925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a312375-35ba-4162-a2c0-e3cc52512334 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:31.894379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a312375-35ba-4162-a2c0-e3cc52512334 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:31.894875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a312375-35ba-4162-a2c0-e3cc52512334 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:31.896419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 449/888] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:31 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:31.902116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad568ea-9b44-45b9-a847-162755f84784 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:31.902403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad568ea-9b44-45b9-a847-162755f84784 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:31.902680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad568ea-9b44-45b9-a847-162755f84784 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:31.906010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd7edaa-ecca-4c41-82ce-48f3dee6cdad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:31.906856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd7edaa-ecca-4c41-82ce-48f3dee6cdad None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:31.907283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd7edaa-ecca-4c41-82ce-48f3dee6cdad None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:31.907701 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd7edaa-ecca-4c41-82ce-48f3dee6cdad None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:31.908504 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd7edaa-ecca-4c41-82ce-48f3dee6cdad None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:31.910567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd7edaa-ecca-4c41-82ce-48f3dee6cdad None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:31.916750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 440/889] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Sep 28 12:02:31 2022] GET /identity/v3/services => generated 1268 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:02:31.922730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-cad568ea-9b44-45b9-a847-162755f84784 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:31.924217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-cad568ea-9b44-45b9-a847-162755f84784 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:32.010641 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 450/890] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:31 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:32.069408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0d58f2-6b1a-4395-93f6-7b93603483f7 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:32.070012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0d58f2-6b1a-4395-93f6-7b93603483f7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:32.070478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0d58f2-6b1a-4395-93f6-7b93603483f7 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:32.072088 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 441/891] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:32.078214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df2d92c-aab1-4ef6-99bc-56b11987f8af None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:32.078794 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df2d92c-aab1-4ef6-99bc-56b11987f8af None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:32.079252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df2d92c-aab1-4ef6-99bc-56b11987f8af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:32.099350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-6df2d92c-aab1-4ef6-99bc-56b11987f8af None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:32.100919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6df2d92c-aab1-4ef6-99bc-56b11987f8af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:32.134248 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:32.176769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 451/892] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:32 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:32.229028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfbf8f4-4604-4eb7-8be7-05f8ee4a803a None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:32.230015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfbf8f4-4604-4eb7-8be7-05f8ee4a803a None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:32.230503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfbf8f4-4604-4eb7-8be7-05f8ee4a803a None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:32.232371 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 442/893] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:32.237418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb6d1d1-b046-4f0a-932b-36ce65d7437b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:32.238179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb6d1d1-b046-4f0a-932b-36ce65d7437b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:32.238458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb6d1d1-b046-4f0a-932b-36ce65d7437b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:32.256418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-1cb6d1d1-b046-4f0a-932b-36ce65d7437b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:32.257606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1cb6d1d1-b046-4f0a-932b-36ce65d7437b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:32.290438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfbf8f4-4604-4eb7-8be7-05f8ee4a803a None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:32.330057 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 452/894] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:32 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:32.341532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:32.399374 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfbf8f4-4604-4eb7-8be7-05f8ee4a803a None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:32.434755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4eae419-b9dc-4c76-8da0-aa89d6aa0b36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:32.435349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eae419-b9dc-4c76-8da0-aa89d6aa0b36 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:32.435629 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eae419-b9dc-4c76-8da0-aa89d6aa0b36 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:32.435896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eae419-b9dc-4c76-8da0-aa89d6aa0b36 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:32.436509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4eae419-b9dc-4c76-8da0-aa89d6aa0b36 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:32.437889 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4eae419-b9dc-4c76-8da0-aa89d6aa0b36 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-c4eae419-b9dc-4c76-8da0-aa89d6aa0b36 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:32.447003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:32.447951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:32.448901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 443/895] 10.222.0.26 () {56 vars in 1132 bytes} [Wed Sep 28 12:02:32 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:32.456792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb6d1d1-b046-4f0a-932b-36ce65d7437b None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:32.478116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4eae419-b9dc-4c76-8da0-aa89d6aa0b36 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:32.585092 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:32.589004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4eae419-b9dc-4c76-8da0-aa89d6aa0b36 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:32.590694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:32.595562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb6d1d1-b046-4f0a-932b-36ce65d7437b None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:32.625249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c4d36a-0501-4fea-8d5c-30aa5ad3f9da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:32.625904 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c4d36a-0501-4fea-8d5c-30aa5ad3f9da None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:32.626187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c4d36a-0501-4fea-8d5c-30aa5ad3f9da None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:32.626461 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c4d36a-0501-4fea-8d5c-30aa5ad3f9da None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:32.634914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41948149-aa08-4d3b-99de-351439f82ea8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:32.635715 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41948149-aa08-4d3b-99de-351439f82ea8 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:32.636144 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41948149-aa08-4d3b-99de-351439f82ea8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:32.636847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41948149-aa08-4d3b-99de-351439f82ea8 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:32.637156 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c4d36a-0501-4fea-8d5c-30aa5ad3f9da None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:32.637788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41948149-aa08-4d3b-99de-351439f82ea8 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:32.638259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41948149-aa08-4d3b-99de-351439f82ea8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:32.639757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41948149-aa08-4d3b-99de-351439f82ea8 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-39c4d36a-0501-4fea-8d5c-30aa5ad3f9da None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:32.642347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:32.643451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:32.645271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:32.645271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:32.645271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:32.645271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:32.645271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:32.645271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:32.645271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:32.645271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:32.645271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:32.645271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:32.645271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 28 12:02:32.645271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:32.645271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 444/896] 10.222.0.26 () {56 vars in 1120 bytes} [Wed Sep 28 12:02:32 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:32.645271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c4d36a-0501-4fea-8d5c-30aa5ad3f9da None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:32.656509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 453/897] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Sep 28 12:02:32 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:32.656509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41948149-aa08-4d3b-99de-351439f82ea8 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:32.751342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c4d36a-0501-4fea-8d5c-30aa5ad3f9da None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:32.761741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41948149-aa08-4d3b-99de-351439f82ea8 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:32.785045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072dba80-441d-4b91-9c2e-cebccd98950e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:32.785714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072dba80-441d-4b91-9c2e-cebccd98950e None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:32.785999 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072dba80-441d-4b91-9c2e-cebccd98950e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:32.786340 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072dba80-441d-4b91-9c2e-cebccd98950e None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:32.786991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072dba80-441d-4b91-9c2e-cebccd98950e None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:32.788387 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072dba80-441d-4b91-9c2e-cebccd98950e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:32.797114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b984d59-45ff-4540-968f-af6566d781b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:32.797777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b984d59-45ff-4540-968f-af6566d781b6 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:32.798057 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b984d59-45ff-4540-968f-af6566d781b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:32.798328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b984d59-45ff-4540-968f-af6566d781b6 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:32.799044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b984d59-45ff-4540-968f-af6566d781b6 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:32.799360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b984d59-45ff-4540-968f-af6566d781b6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:32.800792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b984d59-45ff-4540-968f-af6566d781b6 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:32.803541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 445/898] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Sep 28 12:02:32 2022] POST /identity/v3/services => generated 241 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:32.810318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072dba80-441d-4b91-9c2e-cebccd98950e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:32.818737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 454/899] 10.222.0.26 () {56 vars in 1115 bytes} [Wed Sep 28 12:02:32 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:32.820189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b984d59-45ff-4540-968f-af6566d781b6 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:32.843174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:32.847651 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:32.936449 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b984d59-45ff-4540-968f-af6566d781b6 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:32.937704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072dba80-441d-4b91-9c2e-cebccd98950e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:32.975487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f8b7ab-fbe4-43c3-b899-bdf9137dcea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:32.975853 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb2542e-cb7f-4b1f-a972-f9be2127f25e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:32.976511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb2542e-cb7f-4b1f-a972-f9be2127f25e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:32.976789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb2542e-cb7f-4b1f-a972-f9be2127f25e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:32.977057 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb2542e-cb7f-4b1f-a972-f9be2127f25e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:32.977482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f8b7ab-fbe4-43c3-b899-bdf9137dcea9 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:32.978132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f8b7ab-fbe4-43c3-b899-bdf9137dcea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:32.978621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f8b7ab-fbe4-43c3-b899-bdf9137dcea9 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:32.979495 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f8b7ab-fbe4-43c3-b899-bdf9137dcea9 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:32.980076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f8b7ab-fbe4-43c3-b899-bdf9137dcea9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:32.981965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f8b7ab-fbe4-43c3-b899-bdf9137dcea9 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:32.988573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 446/900] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Sep 28 12:02:32 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:32.989188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb2542e-cb7f-4b1f-a972-f9be2127f25e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:32.990801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb2542e-cb7f-4b1f-a972-f9be2127f25e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:32.994407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f8b7ab-fbe4-43c3-b899-bdf9137dcea9 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-5fb2542e-cb7f-4b1f-a972-f9be2127f25e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:33.007407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:33.008509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:33.009402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:33.009402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:33.009402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:33.012370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 455/901] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:33.015750 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb2542e-cb7f-4b1f-a972-f9be2127f25e None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:33.098209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:33.112244 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:33.133137 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb2542e-cb7f-4b1f-a972-f9be2127f25e None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:33.143218 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f8b7ab-fbe4-43c3-b899-bdf9137dcea9 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:33.170814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d544406d-3242-4438-bf2d-295d804be925 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:33.171425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d544406d-3242-4438-bf2d-295d804be925 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:33.171759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d544406d-3242-4438-bf2d-295d804be925 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:33.172382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d544406d-3242-4438-bf2d-295d804be925 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:33.172781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d544406d-3242-4438-bf2d-295d804be925 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:33.173083 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d544406d-3242-4438-bf2d-295d804be925 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:33.174444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d544406d-3242-4438-bf2d-295d804be925 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:33.189465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25d762a-6ec8-423d-9408-d636a0947637 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:33.190460 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d762a-6ec8-423d-9408-d636a0947637 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:33.190951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d762a-6ec8-423d-9408-d636a0947637 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:33.191430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d762a-6ec8-423d-9408-d636a0947637 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:33.192021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 456/902] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:33.192687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25d762a-6ec8-423d-9408-d636a0947637 None admin] RBAC: Adding query filter params (interface=public, service_id=f47dbc730f0547c2a48e89f06288b34c, region_id=RegionOne) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:33.193255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25d762a-6ec8-423d-9408-d636a0947637 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:33.193362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d544406d-3242-4438-bf2d-295d804be925 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:33.195103 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25d762a-6ec8-423d-9408-d636a0947637 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:33.208829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 447/903] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Sep 28 12:02:32 2022] GET /identity/v3/endpoints?service_id=f47dbc730f0547c2a48e89f06288b34c&interface=public®ion_id=RegionOne => generated 194 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:33.210918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25d762a-6ec8-423d-9408-d636a0947637 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:33.316022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d544406d-3242-4438-bf2d-295d804be925 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:33.332650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25d762a-6ec8-423d-9408-d636a0947637 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:33.360189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:33.367818 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1215dc1d-0181-4b01-b9d6-0cabe1b753c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:33.368852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1215dc1d-0181-4b01-b9d6-0cabe1b753c6 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:33.368974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032253b6-dbb7-4a2b-a0b5-1b2b81106abf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:33.369396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1215dc1d-0181-4b01-b9d6-0cabe1b753c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:33.369721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032253b6-dbb7-4a2b-a0b5-1b2b81106abf None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:33.370041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1215dc1d-0181-4b01-b9d6-0cabe1b753c6 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:33.370165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032253b6-dbb7-4a2b-a0b5-1b2b81106abf None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:33.370542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032253b6-dbb7-4a2b-a0b5-1b2b81106abf None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:33.371241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032253b6-dbb7-4a2b-a0b5-1b2b81106abf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:33.372730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032253b6-dbb7-4a2b-a0b5-1b2b81106abf None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:33.378269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1215dc1d-0181-4b01-b9d6-0cabe1b753c6 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:33.384335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 448/904] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Sep 28 12:02:33 2022] GET /identity/v3/services => generated 1497 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:02:33.384543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1215dc1d-0181-4b01-b9d6-0cabe1b753c6 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-1215dc1d-0181-4b01-b9d6-0cabe1b753c6 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:33.401708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:33.402760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:33.403686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:33.403686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:33.403686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:33.403686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:33.403686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:33.403686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:33.403686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:33.403686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:33.403686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 28 12:02:33.403686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:33.407872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 457/905] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Sep 28 12:02:33 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:33.410052 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032253b6-dbb7-4a2b-a0b5-1b2b81106abf None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:33.424139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:33.537120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032253b6-dbb7-4a2b-a0b5-1b2b81106abf None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:33.575687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdacc99-c175-4e76-8a3f-8541cf9f819f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:33.576231 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdacc99-c175-4e76-8a3f-8541cf9f819f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:33.576544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdacc99-c175-4e76-8a3f-8541cf9f819f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:33.576820 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdacc99-c175-4e76-8a3f-8541cf9f819f None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:33.577574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdacc99-c175-4e76-8a3f-8541cf9f819f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:33.577874 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdacc99-c175-4e76-8a3f-8541cf9f819f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:33.579305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdacc99-c175-4e76-8a3f-8541cf9f819f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:33.600362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 449/906] 10.222.0.26 () {56 vars in 1149 bytes} [Wed Sep 28 12:02:33 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:33.606483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1215dc1d-0181-4b01-b9d6-0cabe1b753c6 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:33.714910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1215dc1d-0181-4b01-b9d6-0cabe1b753c6 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:33.751038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0ec84b-b437-44d5-b976-0698c6d0e9f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:33.752095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0ec84b-b437-44d5-b976-0698c6d0e9f9 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:33.752619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0ec84b-b437-44d5-b976-0698c6d0e9f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:33.753650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0ec84b-b437-44d5-b976-0698c6d0e9f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:33.825271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:33.838099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0ec84b-b437-44d5-b976-0698c6d0e9f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:33.839801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0ec84b-b437-44d5-b976-0698c6d0e9f9 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:33.942607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 458/907] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:33 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 28 12:02:33.948591 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdacc99-c175-4e76-8a3f-8541cf9f819f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:34.016857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:34.055251 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdacc99-c175-4e76-8a3f-8541cf9f819f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:34.090859 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af58f45-8c09-439f-b88b-57339f6f54b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:34.091737 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af58f45-8c09-439f-b88b-57339f6f54b8 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:34.092179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af58f45-8c09-439f-b88b-57339f6f54b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:34.092624 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af58f45-8c09-439f-b88b-57339f6f54b8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:34.103433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af58f45-8c09-439f-b88b-57339f6f54b8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:34.105142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af58f45-8c09-439f-b88b-57339f6f54b8 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-0af58f45-8c09-439f-b88b-57339f6f54b8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:34.125076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:34.126457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:34.127659 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:34.127659 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:34.127659 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:34.134495 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 450/908] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:34.136162 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0ec84b-b437-44d5-b976-0698c6d0e9f9 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:34.141635 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f78ae74-2f70-4825-9c93-d6d58da25751 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:34.141936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f78ae74-2f70-4825-9c93-d6d58da25751 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:34.142191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f78ae74-2f70-4825-9c93-d6d58da25751 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:34.143769 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 451/909] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:34.148425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86df91b4-1bcf-4ead-80c5-53e6a101ea06 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:34.148708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86df91b4-1bcf-4ead-80c5-53e6a101ea06 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:34.149454 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86df91b4-1bcf-4ead-80c5-53e6a101ea06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:34.167611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-86df91b4-1bcf-4ead-80c5-53e6a101ea06 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:34.168927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-86df91b4-1bcf-4ead-80c5-53e6a101ea06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:34.247228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:34.256401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 452/910] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:34 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:34.264063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0ec84b-b437-44d5-b976-0698c6d0e9f9 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:34.299057 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718287af-1f5f-4f6f-8986-951eff36c791 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:34.299663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718287af-1f5f-4f6f-8986-951eff36c791 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:34.300542 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718287af-1f5f-4f6f-8986-951eff36c791 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:34.300820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718287af-1f5f-4f6f-8986-951eff36c791 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:34.301463 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718287af-1f5f-4f6f-8986-951eff36c791 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:34.301766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718287af-1f5f-4f6f-8986-951eff36c791 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:34.303114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718287af-1f5f-4f6f-8986-951eff36c791 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:34.319052 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 459/911] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:34.320434 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86df91b4-1bcf-4ead-80c5-53e6a101ea06 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:34.376880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:34.426053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86df91b4-1bcf-4ead-80c5-53e6a101ea06 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:34.460509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7d8719-2268-4af5-bc49-d2542a1d5ddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:34.461133 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d8719-2268-4af5-bc49-d2542a1d5ddd None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:34.461408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d8719-2268-4af5-bc49-d2542a1d5ddd None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:34.461676 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d8719-2268-4af5-bc49-d2542a1d5ddd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:34.468774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7d8719-2268-4af5-bc49-d2542a1d5ddd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:34.470262 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7d8719-2268-4af5-bc49-d2542a1d5ddd None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-ea7d8719-2268-4af5-bc49-d2542a1d5ddd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:34.494499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:34.496050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:34.497560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:34.497560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:34.497560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:34.497560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:34.497560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:34.497560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:34.497560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:34.497560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:34.502216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 453/912] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:34.502462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718287af-1f5f-4f6f-8986-951eff36c791 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:34.547043 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7d8719-2268-4af5-bc49-d2542a1d5ddd None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:34.612879 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718287af-1f5f-4f6f-8986-951eff36c791 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:34.649375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed8b1b3-43b8-4c0e-aebd-8ef89c41a77c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:34.649903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed8b1b3-43b8-4c0e-aebd-8ef89c41a77c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:34.650177 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed8b1b3-43b8-4c0e-aebd-8ef89c41a77c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:34.650449 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed8b1b3-43b8-4c0e-aebd-8ef89c41a77c None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:34.652490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed8b1b3-43b8-4c0e-aebd-8ef89c41a77c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:34.652801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed8b1b3-43b8-4c0e-aebd-8ef89c41a77c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:34.654209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed8b1b3-43b8-4c0e-aebd-8ef89c41a77c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:34.654814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7d8719-2268-4af5-bc49-d2542a1d5ddd None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:34.665926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:34.667910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 460/913] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:34.674407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed8b1b3-43b8-4c0e-aebd-8ef89c41a77c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:34.693415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:34.699205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57d3a04-3f1e-48b6-9c13-c218b231bcd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:34.700184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57d3a04-3f1e-48b6-9c13-c218b231bcd8 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:34.700557 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57d3a04-3f1e-48b6-9c13-c218b231bcd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:34.700899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57d3a04-3f1e-48b6-9c13-c218b231bcd8 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:34.701551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57d3a04-3f1e-48b6-9c13-c218b231bcd8 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:34.703532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57d3a04-3f1e-48b6-9c13-c218b231bcd8 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-d57d3a04-3f1e-48b6-9c13-c218b231bcd8 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:34.715673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:34.717472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:34.719314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 454/914] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:02:34 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:34.728350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57d3a04-3f1e-48b6-9c13-c218b231bcd8 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:34.790162 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed8b1b3-43b8-4c0e-aebd-8ef89c41a77c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:34.826962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fccfe3f-526c-4b33-b6dd-56cb48462340 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:34.827588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fccfe3f-526c-4b33-b6dd-56cb48462340 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:34.827864 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fccfe3f-526c-4b33-b6dd-56cb48462340 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:34.828136 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fccfe3f-526c-4b33-b6dd-56cb48462340 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:34.828461 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57d3a04-3f1e-48b6-9c13-c218b231bcd8 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:34.828898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fccfe3f-526c-4b33-b6dd-56cb48462340 None admin] RBAC: Adding query filter params (role.id=d1813115b0cf4d2d905502119199b5db, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=b20a6fb4e7a546058f5fa314c9a59299) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:34.832330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fccfe3f-526c-4b33-b6dd-56cb48462340 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:34.832330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fccfe3f-526c-4b33-b6dd-56cb48462340 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:34.837714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 461/915] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:34 2022] GET /identity/v3/role_assignments?user.id=b20a6fb4e7a546058f5fa314c9a59299&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=d1813115b0cf4d2d905502119199b5db => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:34.872477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7803e6f5-3f71-40a4-a2ea-a44c76eee646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:34.873104 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7803e6f5-3f71-40a4-a2ea-a44c76eee646 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:34.873377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7803e6f5-3f71-40a4-a2ea-a44c76eee646 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:34.873646 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7803e6f5-3f71-40a4-a2ea-a44c76eee646 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:34.874356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7803e6f5-3f71-40a4-a2ea-a44c76eee646 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:34.874659 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7803e6f5-3f71-40a4-a2ea-a44c76eee646 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:34.876127 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7803e6f5-3f71-40a4-a2ea-a44c76eee646 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:34.882260 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 455/916] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:02:34 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:34.887777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fccfe3f-526c-4b33-b6dd-56cb48462340 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:34.983582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:35.010485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fccfe3f-526c-4b33-b6dd-56cb48462340 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:35.046545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad75cfe-e34f-466c-8360-cc492651b849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:35.047151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad75cfe-e34f-466c-8360-cc492651b849 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:35.047428 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad75cfe-e34f-466c-8360-cc492651b849 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:35.047701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad75cfe-e34f-466c-8360-cc492651b849 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:35.048389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad75cfe-e34f-466c-8360-cc492651b849 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:35.048695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad75cfe-e34f-466c-8360-cc492651b849 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:35.050087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad75cfe-e34f-466c-8360-cc492651b849 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:35.055456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 462/917] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Sep 28 12:02:34 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:35.061407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7803e6f5-3f71-40a4-a2ea-a44c76eee646 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:35.082822 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47706db8-d605-4574-b355-beb600df0d29 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:35.083146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47706db8-d605-4574-b355-beb600df0d29 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:35.083370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47706db8-d605-4574-b355-beb600df0d29 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:35.084993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 463/918] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:35.090349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27666c6-95f0-4e58-b174-54fc573bc35a None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:35.090636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27666c6-95f0-4e58-b174-54fc573bc35a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:35.090916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27666c6-95f0-4e58-b174-54fc573bc35a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:35.106200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:35.110475 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-f27666c6-95f0-4e58-b174-54fc573bc35a None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:35.111912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f27666c6-95f0-4e58-b174-54fc573bc35a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:35.183723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7803e6f5-3f71-40a4-a2ea-a44c76eee646 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:35.192667 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 464/919] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:35 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:35.219153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a9c1db-8d3a-4880-ad75-33e4433d36c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:35.219846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a9c1db-8d3a-4880-ad75-33e4433d36c4 None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:35.220195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a9c1db-8d3a-4880-ad75-33e4433d36c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:35.220556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a9c1db-8d3a-4880-ad75-33e4433d36c4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:35.221295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a9c1db-8d3a-4880-ad75-33e4433d36c4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:35.223447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a9c1db-8d3a-4880-ad75-33e4433d36c4 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:35.250714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 456/920] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:02:35 2022] POST /identity/v3/endpoints => generated 346 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:35.464066 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27666c6-95f0-4e58-b174-54fc573bc35a None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:35.541985 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:35.576808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27666c6-95f0-4e58-b174-54fc573bc35a None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:35.618451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cf93e9-684c-4dd3-aee0-3e2a4d8c5641 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:35.619099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cf93e9-684c-4dd3-aee0-3e2a4d8c5641 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:35.619377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cf93e9-684c-4dd3-aee0-3e2a4d8c5641 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:35.619654 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cf93e9-684c-4dd3-aee0-3e2a4d8c5641 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:35.620252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cf93e9-684c-4dd3-aee0-3e2a4d8c5641 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:35.621891 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cf93e9-684c-4dd3-aee0-3e2a4d8c5641 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-20cf93e9-684c-4dd3-aee0-3e2a4d8c5641 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:35.632572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:35.633685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:35.634533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 465/921] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Sep 28 12:02:35 2022] GET /identity/v3/services/object-store => generated 93 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:35.640247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a9c1db-8d3a-4880-ad75-33e4433d36c4 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:35.744773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:35.751754 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2d6e35-1ac4-4e77-8cd8-7db2f1908409 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:35.752104 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2d6e35-1ac4-4e77-8cd8-7db2f1908409 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:35.752773 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2d6e35-1ac4-4e77-8cd8-7db2f1908409 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:35.755548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 466/922] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:35.757480 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a9c1db-8d3a-4880-ad75-33e4433d36c4 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:35.759841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45add0e6-e113-4d5d-add3-63dd49ae0295 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:35.760127 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45add0e6-e113-4d5d-add3-63dd49ae0295 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:35.760558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45add0e6-e113-4d5d-add3-63dd49ae0295 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:35.786661 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-45add0e6-e113-4d5d-add3-63dd49ae0295 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:35.788270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-45add0e6-e113-4d5d-add3-63dd49ae0295 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:35.801514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a61e55e-9981-4823-88fe-71f057d2261b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:35.802327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a61e55e-9981-4823-88fe-71f057d2261b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:35.802449 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a61e55e-9981-4823-88fe-71f057d2261b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:35.802648 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a61e55e-9981-4823-88fe-71f057d2261b None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:35.803291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a61e55e-9981-4823-88fe-71f057d2261b None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:35.803596 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a61e55e-9981-4823-88fe-71f057d2261b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:35.804995 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a61e55e-9981-4823-88fe-71f057d2261b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:35.810071 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 457/923] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Sep 28 12:02:35 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:35.816106 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a61e55e-9981-4823-88fe-71f057d2261b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:35.880557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 467/924] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:35 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:35.943778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a61e55e-9981-4823-88fe-71f057d2261b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:35.983829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbd0aac-0eeb-4486-92a1-9219dc340dcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:35.984469 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbd0aac-0eeb-4486-92a1-9219dc340dcc None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:35.984756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbd0aac-0eeb-4486-92a1-9219dc340dcc None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:35.985030 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbd0aac-0eeb-4486-92a1-9219dc340dcc None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:35.985663 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfbd0aac-0eeb-4486-92a1-9219dc340dcc None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:35.985968 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfbd0aac-0eeb-4486-92a1-9219dc340dcc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:35.987943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfbd0aac-0eeb-4486-92a1-9219dc340dcc None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:35.993353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 458/925] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Sep 28 12:02:35 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:35.999113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45add0e6-e113-4d5d-add3-63dd49ae0295 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:36.059113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:36.100619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45add0e6-e113-4d5d-add3-63dd49ae0295 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:36.135309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8559bc-42fe-4763-88fd-cad70d03d9da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:36.135913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8559bc-42fe-4763-88fd-cad70d03d9da None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:36.136189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8559bc-42fe-4763-88fd-cad70d03d9da None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:36.136491 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8559bc-42fe-4763-88fd-cad70d03d9da None admin] PATH_INFO: `/v3/endpoints` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:36.137190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8559bc-42fe-4763-88fd-cad70d03d9da None admin] RBAC: Adding query filter params (interface=public, service_id=d7bf3bfb16d94e7bb91dd60826a9ca1f, region_id=RegionOne) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:36.137495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8559bc-42fe-4763-88fd-cad70d03d9da None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:36.138840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8559bc-42fe-4763-88fd-cad70d03d9da None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:36.148413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 468/926] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Sep 28 12:02:35 2022] GET /identity/v3/endpoints?service_id=d7bf3bfb16d94e7bb91dd60826a9ca1f&interface=public®ion_id=RegionOne => generated 194 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:36.150514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbd0aac-0eeb-4486-92a1-9219dc340dcc None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:36.175527 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:36.183391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8559bc-42fe-4763-88fd-cad70d03d9da None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:36.276223 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbd0aac-0eeb-4486-92a1-9219dc340dcc None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:36.320064 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8559bc-42fe-4763-88fd-cad70d03d9da None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:36.320978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21bd881-f66c-44b7-a6a5-55e19aa54fe9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:36.321751 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21bd881-f66c-44b7-a6a5-55e19aa54fe9 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:36.322027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21bd881-f66c-44b7-a6a5-55e19aa54fe9 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:36.322295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21bd881-f66c-44b7-a6a5-55e19aa54fe9 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:36.322896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21bd881-f66c-44b7-a6a5-55e19aa54fe9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:36.324891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21bd881-f66c-44b7-a6a5-55e19aa54fe9 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:36.336446 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 459/927] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Sep 28 12:02:36 2022] GET /identity/v3/services => generated 1497 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:02:36.350357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:36.365224 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c36d32-d8d2-4177-8d6c-efa6f39bcf18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:36.366263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c36d32-d8d2-4177-8d6c-efa6f39bcf18 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:36.366741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c36d32-d8d2-4177-8d6c-efa6f39bcf18 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:36.367232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c36d32-d8d2-4177-8d6c-efa6f39bcf18 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:36.368093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c36d32-d8d2-4177-8d6c-efa6f39bcf18 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:36.369710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c36d32-d8d2-4177-8d6c-efa6f39bcf18 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-59c36d32-d8d2-4177-8d6c-efa6f39bcf18 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:36.380301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:36.381359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:36.382227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:36.382227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:36.382227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:36.382227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:36.382227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:36.382227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:36.382227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:36.382227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:36.382227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:36.382227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:36.382227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:36.382227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:36.382227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 28 12:02:36.382227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:36.385133 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 469/928] 10.222.0.26 () {56 vars in 1132 bytes} [Wed Sep 28 12:02:36 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:36.391438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21bd881-f66c-44b7-a6a5-55e19aa54fe9 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:36.488226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:36.507171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21bd881-f66c-44b7-a6a5-55e19aa54fe9 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:36.542301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37765ed3-0ea8-413d-b0e6-e1ceb6ec2c32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:36.542905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37765ed3-0ea8-413d-b0e6-e1ceb6ec2c32 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:36.543180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37765ed3-0ea8-413d-b0e6-e1ceb6ec2c32 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:36.543447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37765ed3-0ea8-413d-b0e6-e1ceb6ec2c32 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:36.544078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37765ed3-0ea8-413d-b0e6-e1ceb6ec2c32 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:36.544401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37765ed3-0ea8-413d-b0e6-e1ceb6ec2c32 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:36.546006 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37765ed3-0ea8-413d-b0e6-e1ceb6ec2c32 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:36.551890 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 460/929] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Sep 28 12:02:36 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:36.557476 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c36d32-d8d2-4177-8d6c-efa6f39bcf18 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:36.664152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c36d32-d8d2-4177-8d6c-efa6f39bcf18 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:36.700528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f47b6d-4931-49c3-b8f0-b45dde33419d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:36.701134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f47b6d-4931-49c3-b8f0-b45dde33419d None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:36.701411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f47b6d-4931-49c3-b8f0-b45dde33419d None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:36.701683 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f47b6d-4931-49c3-b8f0-b45dde33419d None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:36.702312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f47b6d-4931-49c3-b8f0-b45dde33419d None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:36.702616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f47b6d-4931-49c3-b8f0-b45dde33419d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:36.704028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f47b6d-4931-49c3-b8f0-b45dde33419d None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:36.709336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 470/930] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Sep 28 12:02:36 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:36.715308 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37765ed3-0ea8-413d-b0e6-e1ceb6ec2c32 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:36.819579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37765ed3-0ea8-413d-b0e6-e1ceb6ec2c32 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:36.854762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497b7e3e-bdfc-4296-b5a1-155f3843e42d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:36.855402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497b7e3e-bdfc-4296-b5a1-155f3843e42d None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:36.855647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497b7e3e-bdfc-4296-b5a1-155f3843e42d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:36.855919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497b7e3e-bdfc-4296-b5a1-155f3843e42d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:36.856615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497b7e3e-bdfc-4296-b5a1-155f3843e42d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:36.857992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497b7e3e-bdfc-4296-b5a1-155f3843e42d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:36.880969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 461/931] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:02:36 2022] POST /identity/v3/endpoints => generated 347 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:37.206710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad276ca9-d4d5-474d-a8e4-c2cad0a3089b None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:37.207260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad276ca9-d4d5-474d-a8e4-c2cad0a3089b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:37.207260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad276ca9-d4d5-474d-a8e4-c2cad0a3089b None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:37.208876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 471/932] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:37.214233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37672196-e923-4f74-9b26-166a5a1e7ef1 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:37.214524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37672196-e923-4f74-9b26-166a5a1e7ef1 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:37.214801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37672196-e923-4f74-9b26-166a5a1e7ef1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:37.228005 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:37.234713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-37672196-e923-4f74-9b26-166a5a1e7ef1 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:37.236173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-37672196-e923-4f74-9b26-166a5a1e7ef1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:37.319203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 462/933] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:37 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:37.547909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3869fcac-6ad1-4b08-9356-2c418763748a None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:37.548232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3869fcac-6ad1-4b08-9356-2c418763748a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:37.548515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3869fcac-6ad1-4b08-9356-2c418763748a None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:37.550669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 472/934] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:37.555952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4510add-8a24-4f4b-8e56-0f5be379438f None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:37.556237 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4510add-8a24-4f4b-8e56-0f5be379438f None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:37.556550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4510add-8a24-4f4b-8e56-0f5be379438f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:37.574570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-b4510add-8a24-4f4b-8e56-0f5be379438f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:37.575754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b4510add-8a24-4f4b-8e56-0f5be379438f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:37.621090 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3869fcac-6ad1-4b08-9356-2c418763748a None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:37.653366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 463/935] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:37 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:37.685462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:37.739519 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3869fcac-6ad1-4b08-9356-2c418763748a None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:37.778756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8e729a-e2c2-4d26-bb88-9047dba39d50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:37.779650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8e729a-e2c2-4d26-bb88-9047dba39d50 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:37.780122 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8e729a-e2c2-4d26-bb88-9047dba39d50 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:37.780616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8e729a-e2c2-4d26-bb88-9047dba39d50 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:37.791597 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8e729a-e2c2-4d26-bb88-9047dba39d50 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-bd8e729a-e2c2-4d26-bb88-9047dba39d50 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:37.796588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:37.797493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:37.798324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:37.798324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:37.798324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:37.798324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:37.798324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:37.798324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:37.798324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:37.798324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:37.798324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:37.798324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:37.798324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 28 12:02:37.798324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:37.800877 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 473/936] 10.222.0.26 () {56 vars in 1120 bytes} [Wed Sep 28 12:02:37 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:37.809825 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4510add-8a24-4f4b-8e56-0f5be379438f None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:37.880678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:37.914373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4510add-8a24-4f4b-8e56-0f5be379438f None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:37.946594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8e729a-e2c2-4d26-bb88-9047dba39d50 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:37.950432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f98e76f-0c79-439e-86f5-504dfd4b9321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:37.951039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f98e76f-0c79-439e-86f5-504dfd4b9321 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:37.951313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f98e76f-0c79-439e-86f5-504dfd4b9321 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:37.951583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f98e76f-0c79-439e-86f5-504dfd4b9321 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:37.953189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f98e76f-0c79-439e-86f5-504dfd4b9321 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:37.953497 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f98e76f-0c79-439e-86f5-504dfd4b9321 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:37.954922 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f98e76f-0c79-439e-86f5-504dfd4b9321 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:37.965447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 464/937] 10.222.0.26 () {56 vars in 1115 bytes} [Wed Sep 28 12:02:37 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:37.970547 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f98e76f-0c79-439e-86f5-504dfd4b9321 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:38.071592 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8e729a-e2c2-4d26-bb88-9047dba39d50 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:38.091007 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:38.112938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f98e76f-0c79-439e-86f5-504dfd4b9321 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:38.122726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6962949c-3556-453b-b518-8419522ea14c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:38.123674 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6962949c-3556-453b-b518-8419522ea14c None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:38.124663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6962949c-3556-453b-b518-8419522ea14c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:38.125156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6962949c-3556-453b-b518-8419522ea14c None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:38.126456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6962949c-3556-453b-b518-8419522ea14c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:38.128472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6962949c-3556-453b-b518-8419522ea14c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:38.131596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: WARNING keystone.server.flask.common [None req-6962949c-3556-453b-b518-8419522ea14c None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 28 12:02:38.140876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:38.153383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1134a9b-97fa-4d0f-b8a1-866452dc66e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:38.154084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1134a9b-97fa-4d0f-b8a1-866452dc66e7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:38.154439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1134a9b-97fa-4d0f-b8a1-866452dc66e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:38.154785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1134a9b-97fa-4d0f-b8a1-866452dc66e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:38.156970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.sql.core [None req-6962949c-3556-453b-b518-8419522ea14c None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 28 12:02:38.156970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 28 12:02:38.156970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [parameters: {'id': '7c0ef0c132de4e1ba93c7d9f1b64568a', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 28 12:02:38.156970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81593) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 28 12:02:38.166945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1134a9b-97fa-4d0f-b8a1-866452dc66e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:38.168549 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1134a9b-97fa-4d0f-b8a1-866452dc66e7 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:38.178763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-6962949c-3556-453b-b518-8419522ea14c None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 7c0ef0c132de4e1ba93c7d9f1b64568a.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 7c0ef0c132de4e1ba93c7d9f1b64568a. Sep 28 12:02:38.178763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:38.178763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 28 12:02:38.178763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 28 12:02:38.178763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 28 12:02:38.178763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 28 12:02:38.178763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 28 12:02:38.178763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application result = self._query(query) Sep 28 12:02:38.178763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 28 12:02:38.178763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application conn.query(q) Sep 28 12:02:38.178763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 28 12:02:38.178763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 28 12:02:38.178763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application result.read() Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 28 12:02:38.179694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application next(self.gen) Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application yield resource Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application next(self.gen) Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self.session.rollback() Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self.force_reraise() Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise self.value Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application session.commit() Sep 28 12:02:38.180717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._prepare_impl() Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self.session.flush() Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._flush(objects) Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application compat.raise_( Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application flush_context.execute() Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 28 12:02:38.181594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rec.execute(self) Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 28 12:02:38.182486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application result = self._query(query) Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application conn.query(q) Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application result.read() Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 28 12:02:38.183282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [parameters: {'id': '7c0ef0c132de4e1ba93c7d9f1b64568a', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:38.184154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:38.186535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:38.188517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:38.188517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 28 12:02:38.188517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 28 12:02:38.188517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:38.188517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:38.188517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 28 12:02:38.188517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 28 12:02:38.188517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 7c0ef0c132de4e1ba93c7d9f1b64568a. Sep 28 12:02:38.188517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:38.188517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 474/938] 10.222.0.26 () {60 vars in 1140 bytes} [Wed Sep 28 12:02:37 2022] POST /identity/v3/projects => generated 262 bytes in 234 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:38.189700 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6962949c-3556-453b-b518-8419522ea14c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-c1134a9b-97fa-4d0f-b8a1-866452dc66e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:38.195346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:38.196249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:38.197226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:38.197226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:38.197226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:38.199493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 465/939] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:38.205279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1134a9b-97fa-4d0f-b8a1-866452dc66e7 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:38.310095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6962949c-3556-453b-b518-8419522ea14c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:38.328723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1134a9b-97fa-4d0f-b8a1-866452dc66e7 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:38.349120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0ad2ad-aba1-47de-85ed-184f8caca16d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:38.349949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0ad2ad-aba1-47de-85ed-184f8caca16d None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:38.350337 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0ad2ad-aba1-47de-85ed-184f8caca16d None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:38.350716 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0ad2ad-aba1-47de-85ed-184f8caca16d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:38.358169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0ad2ad-aba1-47de-85ed-184f8caca16d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:38.360195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0ad2ad-aba1-47de-85ed-184f8caca16d None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:38.364684 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10091498-958e-44c3-87a5-65b86a289edf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:38.365286 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10091498-958e-44c3-87a5-65b86a289edf None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:38.365565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10091498-958e-44c3-87a5-65b86a289edf None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:38.365836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10091498-958e-44c3-87a5-65b86a289edf None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:38.366473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10091498-958e-44c3-87a5-65b86a289edf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:38.366769 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10091498-958e-44c3-87a5-65b86a289edf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:38.368114 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10091498-958e-44c3-87a5-65b86a289edf None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-2a0ad2ad-aba1-47de-85ed-184f8caca16d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:38.374141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:38.375232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:38.376358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:38.376358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:38.376358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:38.376358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:38.376358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:38.376358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:38.376358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:38.376358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:38.383776 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 475/940] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:38.384276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 466/941] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:38.385173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0ad2ad-aba1-47de-85ed-184f8caca16d None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:38.391374 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10091498-958e-44c3-87a5-65b86a289edf None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:38.423490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:38.431084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:38.510404 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10091498-958e-44c3-87a5-65b86a289edf None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:38.519300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0ad2ad-aba1-47de-85ed-184f8caca16d None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:38.546894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14202eb1-96e6-45b2-863c-13bd9f51fcc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:38.547492 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14202eb1-96e6-45b2-863c-13bd9f51fcc7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:38.547774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14202eb1-96e6-45b2-863c-13bd9f51fcc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:38.548048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14202eb1-96e6-45b2-863c-13bd9f51fcc7 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:38.555543 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14202eb1-96e6-45b2-863c-13bd9f51fcc7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:38.557049 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14202eb1-96e6-45b2-863c-13bd9f51fcc7 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:38.564797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb04f3d-fc89-4554-9ed7-e8cec69b47c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:38.565702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb04f3d-fc89-4554-9ed7-e8cec69b47c4 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:38.566176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb04f3d-fc89-4554-9ed7-e8cec69b47c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:38.566642 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb04f3d-fc89-4554-9ed7-e8cec69b47c4 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:38.567512 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb04f3d-fc89-4554-9ed7-e8cec69b47c4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:38.568032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb04f3d-fc89-4554-9ed7-e8cec69b47c4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-14202eb1-96e6-45b2-863c-13bd9f51fcc7 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:38.569229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:38.570119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:38.571038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:38.571038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:38.571038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:38.571038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:38.571038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:38.571038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:38.571038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:38.571038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:38.571038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 28 12:02:38.571038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:38.572565 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb04f3d-fc89-4554-9ed7-e8cec69b47c4 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:38.575129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 467/942] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Sep 28 12:02:38 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:38.576042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14202eb1-96e6-45b2-863c-13bd9f51fcc7 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:38.604524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 476/943] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Sep 28 12:02:38 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:38.692707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:38.700089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14202eb1-96e6-45b2-863c-13bd9f51fcc7 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:38.737044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352014cf-f577-4f68-8b55-58262ae42b68 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:38.737616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352014cf-f577-4f68-8b55-58262ae42b68 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:38.738083 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352014cf-f577-4f68-8b55-58262ae42b68 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:38.740030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 477/944] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:38.740736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718d692b-9918-4e9e-969c-f6fed9f5aaa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:38.741370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d692b-9918-4e9e-969c-f6fed9f5aaa5 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:38.741655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d692b-9918-4e9e-969c-f6fed9f5aaa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:38.741933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d692b-9918-4e9e-969c-f6fed9f5aaa5 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:38.742694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718d692b-9918-4e9e-969c-f6fed9f5aaa5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:38.743001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718d692b-9918-4e9e-969c-f6fed9f5aaa5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:38.747726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718d692b-9918-4e9e-969c-f6fed9f5aaa5 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:38.750058 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9b4a55-1901-4a33-b708-b8ee6c8d1906 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:38.750292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9b4a55-1901-4a33-b708-b8ee6c8d1906 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:38.750569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9b4a55-1901-4a33-b708-b8ee6c8d1906 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:38.771012 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-7f9b4a55-1901-4a33-b708-b8ee6c8d1906 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:38.771611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 468/945] 10.222.0.26 () {56 vars in 1149 bytes} [Wed Sep 28 12:02:38 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:38.773227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7f9b4a55-1901-4a33-b708-b8ee6c8d1906 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:38.777036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718d692b-9918-4e9e-969c-f6fed9f5aaa5 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:38.869519 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 478/946] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:38 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:38.888363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718d692b-9918-4e9e-969c-f6fed9f5aaa5 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:38.925761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda1c74a-8c49-4dce-a449-fb1f40761e02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:38.926386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda1c74a-8c49-4dce-a449-fb1f40761e02 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:38.926663 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda1c74a-8c49-4dce-a449-fb1f40761e02 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:38.926930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda1c74a-8c49-4dce-a449-fb1f40761e02 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:38.937072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda1c74a-8c49-4dce-a449-fb1f40761e02 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:38.938533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda1c74a-8c49-4dce-a449-fb1f40761e02 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-fda1c74a-8c49-4dce-a449-fb1f40761e02 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:38.953850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:38.954929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:38.955830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:38.955830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:38.955830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:38.958006 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 469/947] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:38.963539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9b4a55-1901-4a33-b708-b8ee6c8d1906 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:38.985767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:39.076805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9b4a55-1901-4a33-b708-b8ee6c8d1906 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:39.112973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67054adc-d556-421e-b34e-ff7ca3b05bd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:39.113580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67054adc-d556-421e-b34e-ff7ca3b05bd7 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:39.113898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67054adc-d556-421e-b34e-ff7ca3b05bd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:39.114167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67054adc-d556-421e-b34e-ff7ca3b05bd7 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:39.114809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67054adc-d556-421e-b34e-ff7ca3b05bd7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:39.115108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67054adc-d556-421e-b34e-ff7ca3b05bd7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:39.116482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67054adc-d556-421e-b34e-ff7ca3b05bd7 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:39.129312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 479/948] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:39.134521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda1c74a-8c49-4dce-a449-fb1f40761e02 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:39.146935 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67054adc-d556-421e-b34e-ff7ca3b05bd7 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:39.149416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:39.229959 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:39.246853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda1c74a-8c49-4dce-a449-fb1f40761e02 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:39.255981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67054adc-d556-421e-b34e-ff7ca3b05bd7 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:39.283998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31789b09-3191-4a4d-bd8f-fca91cb80936 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:39.284655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31789b09-3191-4a4d-bd8f-fca91cb80936 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:39.284942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31789b09-3191-4a4d-bd8f-fca91cb80936 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:39.285217 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31789b09-3191-4a4d-bd8f-fca91cb80936 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:39.291022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda99a80-c952-4780-9f5c-5fb9ea3d886e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:39.291849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda99a80-c952-4780-9f5c-5fb9ea3d886e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:39.292081 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31789b09-3191-4a4d-bd8f-fca91cb80936 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:39.292555 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda99a80-c952-4780-9f5c-5fb9ea3d886e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:39.293001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda99a80-c952-4780-9f5c-5fb9ea3d886e None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:39.293576 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31789b09-3191-4a4d-bd8f-fca91cb80936 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:39.293948 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda99a80-c952-4780-9f5c-5fb9ea3d886e None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:39.295353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda99a80-c952-4780-9f5c-5fb9ea3d886e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-dda99a80-c952-4780-9f5c-5fb9ea3d886e None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:39.304830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:39.305808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:39.309317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:39.309317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:39.309317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:39.309317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:39.309317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:39.309317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:39.309317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:39.309317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:39.309317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:39.309317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:39.309317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:39.309317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:39.309317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:39.309317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-31789b09-3191-4a4d-bd8f-fca91cb80936 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:39.310050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:39.310831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:39.312679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:39.312679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:39.312679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:39.312679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:39.312679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:39.312679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:39.312679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:39.312679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:39.313178 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 480/949] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Sep 28 12:02:39 2022] GET /identity/v3/services/object-store => generated 93 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:39.313605 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 470/950] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:39.313687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f400d55-1def-4056-9031-06ff3a89f04f None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:39.313687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f400d55-1def-4056-9031-06ff3a89f04f None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:39.313687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f400d55-1def-4056-9031-06ff3a89f04f None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:39.313687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 481/951] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:39 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:39.317064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31789b09-3191-4a4d-bd8f-fca91cb80936 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:39.317370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cba0b28-79a0-464a-99ba-88cc6c88224f None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:39.317652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cba0b28-79a0-464a-99ba-88cc6c88224f None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:39.317919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cba0b28-79a0-464a-99ba-88cc6c88224f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:39.335828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-3cba0b28-79a0-464a-99ba-88cc6c88224f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:39.337058 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-3cba0b28-79a0-464a-99ba-88cc6c88224f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:39.384445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:39.412476 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 482/952] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:39 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:39.412476 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cba0b28-79a0-464a-99ba-88cc6c88224f None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:39.431049 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31789b09-3191-4a4d-bd8f-fca91cb80936 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:39.468999 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d985f09-4bd8-4e7a-b808-71789e4cf40a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:39.469855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d985f09-4bd8-4e7a-b808-71789e4cf40a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:39.470287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d985f09-4bd8-4e7a-b808-71789e4cf40a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:39.470711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d985f09-4bd8-4e7a-b808-71789e4cf40a None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:39.471568 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d985f09-4bd8-4e7a-b808-71789e4cf40a None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:39.472566 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d985f09-4bd8-4e7a-b808-71789e4cf40a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:39.474113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d985f09-4bd8-4e7a-b808-71789e4cf40a None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:39.484369 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 471/953] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Sep 28 12:02:39 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:39.491483 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d985f09-4bd8-4e7a-b808-71789e4cf40a None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:39.510127 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:39.551327 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cba0b28-79a0-464a-99ba-88cc6c88224f None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:39.597675 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5d6a0c-9a41-4bad-af71-6726dda49fe2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:39.598888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5d6a0c-9a41-4bad-af71-6726dda49fe2 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:39.599724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5d6a0c-9a41-4bad-af71-6726dda49fe2 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:39.600484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5d6a0c-9a41-4bad-af71-6726dda49fe2 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:39.601404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5d6a0c-9a41-4bad-af71-6726dda49fe2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:39.601924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5d6a0c-9a41-4bad-af71-6726dda49fe2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:39.602031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d985f09-4bd8-4e7a-b808-71789e4cf40a None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:39.604107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5d6a0c-9a41-4bad-af71-6726dda49fe2 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:39.622292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 483/954] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:39.628227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5d6a0c-9a41-4bad-af71-6726dda49fe2 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:39.638471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:39.646126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7463ac-e2bb-42b4-9408-fa2a523fb098 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:39.646751 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7463ac-e2bb-42b4-9408-fa2a523fb098 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:39.647042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7463ac-e2bb-42b4-9408-fa2a523fb098 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:39.647308 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7463ac-e2bb-42b4-9408-fa2a523fb098 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:39.647960 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7463ac-e2bb-42b4-9408-fa2a523fb098 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:39.648269 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7463ac-e2bb-42b4-9408-fa2a523fb098 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:39.649734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7463ac-e2bb-42b4-9408-fa2a523fb098 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:39.656356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 472/955] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Sep 28 12:02:39 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:39.660961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7463ac-e2bb-42b4-9408-fa2a523fb098 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:39.752743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5d6a0c-9a41-4bad-af71-6726dda49fe2 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:39.766614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7463ac-e2bb-42b4-9408-fa2a523fb098 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:39.784912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:39.791193 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e06bc76-25ac-411c-afb3-67f79cec6a3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:39.791979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e06bc76-25ac-411c-afb3-67f79cec6a3a None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:39.792398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e06bc76-25ac-411c-afb3-67f79cec6a3a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:39.792798 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e06bc76-25ac-411c-afb3-67f79cec6a3a None admin] PATH_INFO: `/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/b20a6fb4e7a546058f5fa314c9a59299/roles/d1813115b0cf4d2d905502119199b5db` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:39.795300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e06bc76-25ac-411c-afb3-67f79cec6a3a None admin] RBAC: Authorizing `identity:create_grant(project_id=8695d562e3e24c7c9300cf3a23cd7502, user_id=b20a6fb4e7a546058f5fa314c9a59299, role_id=d1813115b0cf4d2d905502119199b5db)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:39.796861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e06bc76-25ac-411c-afb3-67f79cec6a3a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:39.802157 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a1e7f1-8b46-4f19-898e-584597f63f7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:39.803152 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a1e7f1-8b46-4f19-898e-584597f63f7b None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:39.803152 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a1e7f1-8b46-4f19-898e-584597f63f7b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:39.803376 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a1e7f1-8b46-4f19-898e-584597f63f7b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:39.804073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a1e7f1-8b46-4f19-898e-584597f63f7b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:39.805600 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a1e7f1-8b46-4f19-898e-584597f63f7b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:39.811656 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 484/956] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:02:39 2022] PUT /identity/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/b20a6fb4e7a546058f5fa314c9a59299/roles/d1813115b0cf4d2d905502119199b5db => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:02:39.812390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e06bc76-25ac-411c-afb3-67f79cec6a3a None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:39.852337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 473/957] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:02:39 2022] POST /identity/v3/endpoints => generated 349 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:02:39.952962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e06bc76-25ac-411c-afb3-67f79cec6a3a None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:40.003547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e908de-c28d-4929-8d24-0b9d7d0e8152 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:40.004585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e908de-c28d-4929-8d24-0b9d7d0e8152 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:40.005009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e908de-c28d-4929-8d24-0b9d7d0e8152 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:40.005465 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e908de-c28d-4929-8d24-0b9d7d0e8152 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:40.006088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e908de-c28d-4929-8d24-0b9d7d0e8152 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:40.007482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e908de-c28d-4929-8d24-0b9d7d0e8152 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-41e908de-c28d-4929-8d24-0b9d7d0e8152 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:40.019726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:40.020951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:40.021845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 485/958] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:39 2022] GET /identity/v3/services/compute => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:40.029475 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a1e7f1-8b46-4f19-898e-584597f63f7b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:40.095328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:40.150145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a1e7f1-8b46-4f19-898e-584597f63f7b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:40.184680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df46490-fd2d-4d1e-a296-73eaf688441a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:40.185287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df46490-fd2d-4d1e-a296-73eaf688441a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:40.185566 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df46490-fd2d-4d1e-a296-73eaf688441a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:40.185834 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df46490-fd2d-4d1e-a296-73eaf688441a None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:40.186465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df46490-fd2d-4d1e-a296-73eaf688441a None admin] RBAC: Adding query filter params (name=compute) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:40.186764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df46490-fd2d-4d1e-a296-73eaf688441a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:40.188182 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df46490-fd2d-4d1e-a296-73eaf688441a None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:40.193463 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 474/959] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Sep 28 12:02:40 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:40.198974 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e908de-c28d-4929-8d24-0b9d7d0e8152 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:40.256720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:40.306349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e908de-c28d-4929-8d24-0b9d7d0e8152 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:40.342456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40192ac1-31c6-4541-9b90-c255af655105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:40.343875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40192ac1-31c6-4541-9b90-c255af655105 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:40.344377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40192ac1-31c6-4541-9b90-c255af655105 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:40.344852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40192ac1-31c6-4541-9b90-c255af655105 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:40.345682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40192ac1-31c6-4541-9b90-c255af655105 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:40.346197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40192ac1-31c6-4541-9b90-c255af655105 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:40.348287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40192ac1-31c6-4541-9b90-c255af655105 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:40.353804 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 486/960] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Sep 28 12:02:40 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:40.978004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e9393b-239b-4c40-be16-f66e80ee2e7d None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:40.978362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e9393b-239b-4c40-be16-f66e80ee2e7d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:40.978669 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e9393b-239b-4c40-be16-f66e80ee2e7d None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:40.980278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 475/961] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:40.985680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbffcdd-12f6-4cf5-9834-2c340f351774 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:40.985973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbffcdd-12f6-4cf5-9834-2c340f351774 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:40.986252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbffcdd-12f6-4cf5-9834-2c340f351774 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:41.005821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-bbbffcdd-12f6-4cf5-9834-2c340f351774 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:41.007003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-bbbffcdd-12f6-4cf5-9834-2c340f351774 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:41.077381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 487/962] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:40 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:41.352238 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e9393b-239b-4c40-be16-f66e80ee2e7d None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:41.455735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e9393b-239b-4c40-be16-f66e80ee2e7d None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:41.469721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:41.491421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a644c9-d609-4815-ad32-2e7aed6cd549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:41.491421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a644c9-d609-4815-ad32-2e7aed6cd549 None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:41.491421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a644c9-d609-4815-ad32-2e7aed6cd549 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:41.491421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a644c9-d609-4815-ad32-2e7aed6cd549 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:41.491968 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a644c9-d609-4815-ad32-2e7aed6cd549 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:41.493446 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a644c9-d609-4815-ad32-2e7aed6cd549 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:41.496553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: WARNING keystone.server.flask.common [None req-e0a644c9-d609-4815-ad32-2e7aed6cd549 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 28 12:02:41.510897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.sql.core [None req-e0a644c9-d609-4815-ad32-2e7aed6cd549 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 28 12:02:41.510897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 28 12:02:41.510897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [parameters: {'user_id': 'ce123c044c594ff6bbbadb2b43e4e9c2', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 28 12:02:41.510897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81592) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-e0a644c9-d609-4815-ad32-2e7aed6cd549 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application result = self._query(query) Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application conn.query(q) Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application result.read() Sep 28 12:02:41.518948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 28 12:02:41.519988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application next(self.gen) Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application yield resource Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application next(self.gen) Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self.session.rollback() Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self.force_reraise() Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise self.value Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application session.commit() Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 28 12:02:41.521209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._prepare_impl() Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self.session.flush() Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._flush(objects) Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application compat.raise_( Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application flush_context.execute() Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rec.execute(self) Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 28 12:02:41.522101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 28 12:02:41.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application result = self._query(query) Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application conn.query(q) Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application result.read() Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 28 12:02:41.524965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [parameters: {'user_id': 'ce123c044c594ff6bbbadb2b43e4e9c2', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:41.525837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 28 12:02:41.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 28 12:02:41.527545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 28 12:02:41.527545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 28 12:02:41.527545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 28 12:02:41.527545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 28 12:02:41.527545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 28 12:02:41.527545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:41.527545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 476/963] 10.222.0.26 () {60 vars in 1131 bytes} [Wed Sep 28 12:02:41 2022] POST /identity/v3/users => generated 160 bytes in 169 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:41.529489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbffcdd-12f6-4cf5-9834-2c340f351774 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:41.584200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:41.639237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbffcdd-12f6-4cf5-9834-2c340f351774 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:41.674350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf338f12-69a5-4b90-aa40-15e26fa1605e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:41.675537 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf338f12-69a5-4b90-aa40-15e26fa1605e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:41.675812 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf338f12-69a5-4b90-aa40-15e26fa1605e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:41.676082 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf338f12-69a5-4b90-aa40-15e26fa1605e None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:41.683368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf338f12-69a5-4b90-aa40-15e26fa1605e None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:41.684871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf338f12-69a5-4b90-aa40-15e26fa1605e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-cf338f12-69a5-4b90-aa40-15e26fa1605e None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:41.696472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:41.697431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:41.698388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:41.698388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:41.698388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:41.698388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:41.698388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:41.698388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:41.698388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:41.698388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:41.698388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 28 12:02:41.698388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:41.698388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 488/964] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:02:41 2022] GET /identity/v3/users/cinder => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:41.703885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a644c9-d609-4815-ad32-2e7aed6cd549 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:41.814280 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a644c9-d609-4815-ad32-2e7aed6cd549 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:41.851920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95beabe9-a16c-46f0-bdf8-98d0b5e77d2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:41.852551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95beabe9-a16c-46f0-bdf8-98d0b5e77d2a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:41.852828 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95beabe9-a16c-46f0-bdf8-98d0b5e77d2a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:41.853097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95beabe9-a16c-46f0-bdf8-98d0b5e77d2a None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:41.853792 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95beabe9-a16c-46f0-bdf8-98d0b5e77d2a None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:41.854094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95beabe9-a16c-46f0-bdf8-98d0b5e77d2a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:41.855721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95beabe9-a16c-46f0-bdf8-98d0b5e77d2a None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:41.893888 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 477/965] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Sep 28 12:02:41 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:42.099145 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5880e3-08c9-4025-a7b8-f1c0c459fb4b None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:42.099425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5880e3-08c9-4025-a7b8-f1c0c459fb4b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:42.099700 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5880e3-08c9-4025-a7b8-f1c0c459fb4b None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:42.101059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 489/966] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:42.106856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf2c0da-e8e0-4534-b145-c3ff636f3799 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:42.107143 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf2c0da-e8e0-4534-b145-c3ff636f3799 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:42.107412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf2c0da-e8e0-4534-b145-c3ff636f3799 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:42.127443 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-2bf2c0da-e8e0-4534-b145-c3ff636f3799 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:42.135890 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2bf2c0da-e8e0-4534-b145-c3ff636f3799 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:42.167421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:42.188564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9227ae40-4ddf-40ed-80cd-c02b9eee7b6e None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:42.188856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9227ae40-4ddf-40ed-80cd-c02b9eee7b6e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:42.189133 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9227ae40-4ddf-40ed-80cd-c02b9eee7b6e None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:42.190799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 490/967] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:42.195980 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2907f5-c3d6-481b-9416-fa83967e774a None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:42.196266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2907f5-c3d6-481b-9416-fa83967e774a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:42.196578 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2907f5-c3d6-481b-9416-fa83967e774a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:42.218059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 478/968] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:42.218552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-de2907f5-c3d6-481b-9416-fa83967e774a None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:42.219893 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-de2907f5-c3d6-481b-9416-fa83967e774a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:42.288519 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 491/969] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:42.501421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf2c0da-e8e0-4534-b145-c3ff636f3799 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:42.559191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb796363-3ed2-4a33-a512-87ffa5f6b73f None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:42.559738 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb796363-3ed2-4a33-a512-87ffa5f6b73f None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:42.559738 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb796363-3ed2-4a33-a512-87ffa5f6b73f None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:42.561213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 492/970] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:42.565078 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb796363-3ed2-4a33-a512-87ffa5f6b73f None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:42.601075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf2c0da-e8e0-4534-b145-c3ff636f3799 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:42.601273 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:42.635008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20987878-1251-4803-b358-b4abf8237e99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:42.635629 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20987878-1251-4803-b358-b4abf8237e99 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:42.635905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20987878-1251-4803-b358-b4abf8237e99 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:42.636173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20987878-1251-4803-b358-b4abf8237e99 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:42.645956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20987878-1251-4803-b358-b4abf8237e99 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-20987878-1251-4803-b358-b4abf8237e99 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:42.650652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:42.651643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:42.652525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:42.652525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:42.652525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:42.652525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:42.652525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:42.652525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:42.652525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:42.652525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:42.652525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:42.652525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:42.652525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 28 12:02:42.652525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:42.652525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 479/971] 10.222.0.26 () {56 vars in 1120 bytes} [Wed Sep 28 12:02:42 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:42.654982 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634c30e5-3f77-44e1-80d5-8a12963430f6 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:42.655265 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634c30e5-3f77-44e1-80d5-8a12963430f6 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:42.655541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634c30e5-3f77-44e1-80d5-8a12963430f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:42.672838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-634c30e5-3f77-44e1-80d5-8a12963430f6 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:42.673184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb796363-3ed2-4a33-a512-87ffa5f6b73f None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:42.674006 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-634c30e5-3f77-44e1-80d5-8a12963430f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:42.691295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:42.707610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714d95cc-14d4-4cdc-9ad3-a839769c552a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:42.708234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714d95cc-14d4-4cdc-9ad3-a839769c552a None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:42.708534 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714d95cc-14d4-4cdc-9ad3-a839769c552a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:42.708809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714d95cc-14d4-4cdc-9ad3-a839769c552a None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:42.709386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714d95cc-14d4-4cdc-9ad3-a839769c552a None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:42.710726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714d95cc-14d4-4cdc-9ad3-a839769c552a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-714d95cc-14d4-4cdc-9ad3-a839769c552a None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:42.719645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:42.720785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:42.722308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 493/972] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Sep 28 12:02:42 2022] GET /identity/v3/services/object-store => generated 93 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:42.723134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714d95cc-14d4-4cdc-9ad3-a839769c552a None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:42.742259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 480/973] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:42.742952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634c30e5-3f77-44e1-80d5-8a12963430f6 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:42.831888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:42.834855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714d95cc-14d4-4cdc-9ad3-a839769c552a None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:42.842707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634c30e5-3f77-44e1-80d5-8a12963430f6 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:42.872027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f8bb68-a3c6-4f58-9f19-32e1e2813b8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:42.872785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f8bb68-a3c6-4f58-9f19-32e1e2813b8b None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:42.873167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f8bb68-a3c6-4f58-9f19-32e1e2813b8b None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:42.873446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f8bb68-a3c6-4f58-9f19-32e1e2813b8b None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:42.874105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f8bb68-a3c6-4f58-9f19-32e1e2813b8b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:42.874409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f8bb68-a3c6-4f58-9f19-32e1e2813b8b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:42.875755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f8bb68-a3c6-4f58-9f19-32e1e2813b8b None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:42.877144 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d04396d-c80c-409b-a43d-d2b2947c4455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:42.877776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d04396d-c80c-409b-a43d-d2b2947c4455 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:42.878057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d04396d-c80c-409b-a43d-d2b2947c4455 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:42.878334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d04396d-c80c-409b-a43d-d2b2947c4455 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:42.878959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d04396d-c80c-409b-a43d-d2b2947c4455 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:42.879266 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d04396d-c80c-409b-a43d-d2b2947c4455 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:42.880651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d04396d-c80c-409b-a43d-d2b2947c4455 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:42.885225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 494/974] 10.222.0.26 () {56 vars in 1115 bytes} [Wed Sep 28 12:02:42 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:42.888342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 481/975] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Sep 28 12:02:42 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:42.891206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d04396d-c80c-409b-a43d-d2b2947c4455 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:42.891324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f8bb68-a3c6-4f58-9f19-32e1e2813b8b None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:42.912729 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:42.998780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d04396d-c80c-409b-a43d-d2b2947c4455 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:43.001999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f8bb68-a3c6-4f58-9f19-32e1e2813b8b None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:43.032416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02cc617-2a7b-4ad2-938f-8b468f39fb49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:43.033017 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02cc617-2a7b-4ad2-938f-8b468f39fb49 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:43.033296 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02cc617-2a7b-4ad2-938f-8b468f39fb49 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:43.033566 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02cc617-2a7b-4ad2-938f-8b468f39fb49 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:43.034864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02cc617-2a7b-4ad2-938f-8b468f39fb49 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:43.035165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02cc617-2a7b-4ad2-938f-8b468f39fb49 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:43.036247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8613b87-df2f-4460-8c17-3968deedbb5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:43.036551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02cc617-2a7b-4ad2-938f-8b468f39fb49 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:43.036900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8613b87-df2f-4460-8c17-3968deedbb5d None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:43.037181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8613b87-df2f-4460-8c17-3968deedbb5d None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:43.037456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8613b87-df2f-4460-8c17-3968deedbb5d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:43.041246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 482/976] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Sep 28 12:02:42 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:43.041997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02cc617-2a7b-4ad2-938f-8b468f39fb49 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:43.049006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8613b87-df2f-4460-8c17-3968deedbb5d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:43.050871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8613b87-df2f-4460-8c17-3968deedbb5d None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-b8613b87-df2f-4460-8c17-3968deedbb5d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:43.064838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:43.065871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:43.066748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:43.066748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:43.066748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:43.066748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 495/977] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:43.066748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8613b87-df2f-4460-8c17-3968deedbb5d None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:43.082228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:43.124745 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:43.143527 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02cc617-2a7b-4ad2-938f-8b468f39fb49 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:43.166725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8613b87-df2f-4460-8c17-3968deedbb5d None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:43.175946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664d06ca-dcf9-4ce0-9f6f-d2a956d7d557 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:43.176571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664d06ca-dcf9-4ce0-9f6f-d2a956d7d557 None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:43.176856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664d06ca-dcf9-4ce0-9f6f-d2a956d7d557 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:43.177140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664d06ca-dcf9-4ce0-9f6f-d2a956d7d557 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:43.177788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664d06ca-dcf9-4ce0-9f6f-d2a956d7d557 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:43.183203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664d06ca-dcf9-4ce0-9f6f-d2a956d7d557 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:43.194450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 483/978] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Sep 28 12:02:43 2022] POST /identity/v3/services => generated 242 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:43.195264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664d06ca-dcf9-4ce0-9f6f-d2a956d7d557 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:43.213719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e95565-ac56-4581-8b2b-558a8dc7ea35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:43.215490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e95565-ac56-4581-8b2b-558a8dc7ea35 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:43.216287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e95565-ac56-4581-8b2b-558a8dc7ea35 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:43.216810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e95565-ac56-4581-8b2b-558a8dc7ea35 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:43.217843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e95565-ac56-4581-8b2b-558a8dc7ea35 None admin] RBAC: Adding query filter params (interface=admin, service_id=d7bf3bfb16d94e7bb91dd60826a9ca1f, region_id=RegionOne) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:43.218352 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e95565-ac56-4581-8b2b-558a8dc7ea35 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:43.219962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e95565-ac56-4581-8b2b-558a8dc7ea35 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:43.227464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 496/979] 10.222.0.26 () {56 vars in 1249 bytes} [Wed Sep 28 12:02:43 2022] GET /identity/v3/endpoints?service_id=d7bf3bfb16d94e7bb91dd60826a9ca1f&interface=admin®ion_id=RegionOne => generated 193 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:43.233655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e95565-ac56-4581-8b2b-558a8dc7ea35 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:43.312711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664d06ca-dcf9-4ce0-9f6f-d2a956d7d557 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:43.318950 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:43.348518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e95565-ac56-4581-8b2b-558a8dc7ea35 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:43.350088 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aef34f2-5c9f-4a81-8f0c-3e370bf88f9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:43.350701 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aef34f2-5c9f-4a81-8f0c-3e370bf88f9c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:43.351716 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aef34f2-5c9f-4a81-8f0c-3e370bf88f9c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:43.351716 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aef34f2-5c9f-4a81-8f0c-3e370bf88f9c None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:43.351906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aef34f2-5c9f-4a81-8f0c-3e370bf88f9c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:43.352178 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aef34f2-5c9f-4a81-8f0c-3e370bf88f9c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:43.353681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aef34f2-5c9f-4a81-8f0c-3e370bf88f9c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:43.366870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 484/980] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:43.372409 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aef34f2-5c9f-4a81-8f0c-3e370bf88f9c None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:43.375981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:43.388604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309b9898-563f-43ec-934c-d4e71d2aaae7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:43.389466 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b9898-563f-43ec-934c-d4e71d2aaae7 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:43.389750 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b9898-563f-43ec-934c-d4e71d2aaae7 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:43.390022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b9898-563f-43ec-934c-d4e71d2aaae7 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:43.390648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309b9898-563f-43ec-934c-d4e71d2aaae7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:43.391995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309b9898-563f-43ec-934c-d4e71d2aaae7 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:43.398556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 497/981] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Sep 28 12:02:43 2022] GET /identity/v3/services => generated 1727 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:02:43.496195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aef34f2-5c9f-4a81-8f0c-3e370bf88f9c None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:43.535393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab3ca16-a7d1-42bd-bc88-4fe2443373d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:43.536094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab3ca16-a7d1-42bd-bc88-4fe2443373d2 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:43.536442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab3ca16-a7d1-42bd-bc88-4fe2443373d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:43.536614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab3ca16-a7d1-42bd-bc88-4fe2443373d2 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:43.544924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab3ca16-a7d1-42bd-bc88-4fe2443373d2 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:43.546389 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab3ca16-a7d1-42bd-bc88-4fe2443373d2 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-4ab3ca16-a7d1-42bd-bc88-4fe2443373d2 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:43.559360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:43.560412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:43.561357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:43.561357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:43.561357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:43.561357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:43.561357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:43.561357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:43.561357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:43.561357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:43.561357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 28 12:02:43.561357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:43.561357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 485/982] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Sep 28 12:02:43 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:43.568371 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309b9898-563f-43ec-934c-d4e71d2aaae7 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:43.671618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309b9898-563f-43ec-934c-d4e71d2aaae7 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:43.704929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd08b27-d0a9-40ff-b49c-37dc1abf44e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:43.705533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd08b27-d0a9-40ff-b49c-37dc1abf44e8 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:43.706331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd08b27-d0a9-40ff-b49c-37dc1abf44e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:43.706600 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd08b27-d0a9-40ff-b49c-37dc1abf44e8 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:43.707324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd08b27-d0a9-40ff-b49c-37dc1abf44e8 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:43.707627 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd08b27-d0a9-40ff-b49c-37dc1abf44e8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:43.709069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd08b27-d0a9-40ff-b49c-37dc1abf44e8 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:43.717886 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:43.733022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 498/983] 10.222.0.26 () {56 vars in 1149 bytes} [Wed Sep 28 12:02:43 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:43.734604 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab3ca16-a7d1-42bd-bc88-4fe2443373d2 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:43.795827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:43.835944 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab3ca16-a7d1-42bd-bc88-4fe2443373d2 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:43.870491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a553882e-911c-4915-86ca-cfbbbb399a02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:43.871099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a553882e-911c-4915-86ca-cfbbbb399a02 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:43.871376 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a553882e-911c-4915-86ca-cfbbbb399a02 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:43.871646 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a553882e-911c-4915-86ca-cfbbbb399a02 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:43.939969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a553882e-911c-4915-86ca-cfbbbb399a02 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:43.941469 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a553882e-911c-4915-86ca-cfbbbb399a02 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:44.001132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:44.045496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 486/984] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:43 2022] GET /identity/v3/auth/tokens => generated 1570 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:02:44.047308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd08b27-d0a9-40ff-b49c-37dc1abf44e8 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:44.151410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd08b27-d0a9-40ff-b49c-37dc1abf44e8 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:44.185378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a8ae39-2752-43e2-8d79-4485f2a20462 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:44.186000 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8ae39-2752-43e2-8d79-4485f2a20462 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:44.186276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8ae39-2752-43e2-8d79-4485f2a20462 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:44.186545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8ae39-2752-43e2-8d79-4485f2a20462 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:44.196525 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a8ae39-2752-43e2-8d79-4485f2a20462 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:44.197895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a8ae39-2752-43e2-8d79-4485f2a20462 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-04a8ae39-2752-43e2-8d79-4485f2a20462 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:44.212232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:44.213479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:44.214376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:44.214376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:44.214376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:44.216631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 499/985] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:44.219318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a553882e-911c-4915-86ca-cfbbbb399a02 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:44.324623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a553882e-911c-4915-86ca-cfbbbb399a02 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:44.357632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfcf3e7-07e1-4356-b2e2-5383ded4a383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:44.358224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfcf3e7-07e1-4356-b2e2-5383ded4a383 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:44.358498 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfcf3e7-07e1-4356-b2e2-5383ded4a383 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:44.358762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfcf3e7-07e1-4356-b2e2-5383ded4a383 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:44.359389 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfcf3e7-07e1-4356-b2e2-5383ded4a383 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:44.359687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfcf3e7-07e1-4356-b2e2-5383ded4a383 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:44.361072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfcf3e7-07e1-4356-b2e2-5383ded4a383 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:44.372178 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 487/986] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:44.377474 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a8ae39-2752-43e2-8d79-4485f2a20462 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:44.426278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:44.481483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a8ae39-2752-43e2-8d79-4485f2a20462 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:44.523748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc155813-2155-4b82-ab93-f54f232c771d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:44.524427 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc155813-2155-4b82-ab93-f54f232c771d None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:44.524705 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc155813-2155-4b82-ab93-f54f232c771d None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:44.524975 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc155813-2155-4b82-ab93-f54f232c771d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:44.531466 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc155813-2155-4b82-ab93-f54f232c771d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:44.532920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc155813-2155-4b82-ab93-f54f232c771d None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-fc155813-2155-4b82-ab93-f54f232c771d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:44.544589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:44.545495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:44.546391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:44.546391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:44.546391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:44.546391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:44.546391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:44.546391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:44.546391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:44.546391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:44.550430 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 500/987] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:44.551414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfcf3e7-07e1-4356-b2e2-5383ded4a383 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:44.576339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:44.651682 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfcf3e7-07e1-4356-b2e2-5383ded4a383 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:44.684856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00b9db1-0212-4cc8-9b5d-b45d493803c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:44.685451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00b9db1-0212-4cc8-9b5d-b45d493803c2 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:44.685721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00b9db1-0212-4cc8-9b5d-b45d493803c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:44.685995 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00b9db1-0212-4cc8-9b5d-b45d493803c2 None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:44.686679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00b9db1-0212-4cc8-9b5d-b45d493803c2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:44.686980 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00b9db1-0212-4cc8-9b5d-b45d493803c2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:44.688431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00b9db1-0212-4cc8-9b5d-b45d493803c2 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:44.700864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 488/988] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:44.706594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc155813-2155-4b82-ab93-f54f232c771d None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:44.803740 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc155813-2155-4b82-ab93-f54f232c771d None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:44.812538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:44.838769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149390a0-f8fa-4de8-8b11-d81d2c9ee165 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:44.839365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149390a0-f8fa-4de8-8b11-d81d2c9ee165 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:44.839638 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149390a0-f8fa-4de8-8b11-d81d2c9ee165 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:44.839959 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149390a0-f8fa-4de8-8b11-d81d2c9ee165 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:44.840705 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149390a0-f8fa-4de8-8b11-d81d2c9ee165 None admin] RBAC: Adding query filter params (role.id=d1813115b0cf4d2d905502119199b5db, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=b20a6fb4e7a546058f5fa314c9a59299) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:44.841009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149390a0-f8fa-4de8-8b11-d81d2c9ee165 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:44.842437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149390a0-f8fa-4de8-8b11-d81d2c9ee165 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:44.856334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 501/989] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:44 2022] GET /identity/v3/role_assignments?user.id=b20a6fb4e7a546058f5fa314c9a59299&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=d1813115b0cf4d2d905502119199b5db => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:45.426690 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711e951f-49f1-4291-9133-66e37d6402d2 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:45.427069 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711e951f-49f1-4291-9133-66e37d6402d2 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:45.427250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711e951f-49f1-4291-9133-66e37d6402d2 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:45.428770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 489/990] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:45.433776 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fbf07e-247b-46c3-9d91-12c45699fbe9 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:45.434070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fbf07e-247b-46c3-9d91-12c45699fbe9 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:45.434349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fbf07e-247b-46c3-9d91-12c45699fbe9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:45.453011 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-37fbf07e-247b-46c3-9d91-12c45699fbe9 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:45.454200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-37fbf07e-247b-46c3-9d91-12c45699fbe9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:45.531686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 502/991] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:45 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:45.590953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfceb51d-21e4-43e3-82ef-9dc2a471ed61 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:45.591490 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfceb51d-21e4-43e3-82ef-9dc2a471ed61 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:45.591490 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfceb51d-21e4-43e3-82ef-9dc2a471ed61 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:45.593024 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 490/992] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:45.598561 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e793c271-7116-469b-bbdc-6686624610f4 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:45.598858 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e793c271-7116-469b-bbdc-6686624610f4 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:45.599138 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e793c271-7116-469b-bbdc-6686624610f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:45.617275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-e793c271-7116-469b-bbdc-6686624610f4 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:45.618462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e793c271-7116-469b-bbdc-6686624610f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:45.686864 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 503/993] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:45 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:45.802447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfceb51d-21e4-43e3-82ef-9dc2a471ed61 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:45.888069 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:45.911738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfceb51d-21e4-43e3-82ef-9dc2a471ed61 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:45.944596 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c08d62-8091-4f09-a0d9-41a0cdbd3673 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:45.945152 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c08d62-8091-4f09-a0d9-41a0cdbd3673 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:45.945432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c08d62-8091-4f09-a0d9-41a0cdbd3673 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:45.945707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c08d62-8091-4f09-a0d9-41a0cdbd3673 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:45.946287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c08d62-8091-4f09-a0d9-41a0cdbd3673 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:45.947657 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c08d62-8091-4f09-a0d9-41a0cdbd3673 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-d5c08d62-8091-4f09-a0d9-41a0cdbd3673 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:45.956842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:45.958094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:45.959125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:45.959125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:45.959125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:45.959125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:45.959125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:45.959125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:45.959125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:45.959125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:45.959125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:45.959125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:45.959125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:45.959125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:45.959125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 28 12:02:45.959125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:45.961145 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e793c271-7116-469b-bbdc-6686624610f4 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:45.963017 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 491/994] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:45 2022] GET /identity/v3/services/compute => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:45.963692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c08d62-8091-4f09-a0d9-41a0cdbd3673 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:45.997112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:46.066738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c08d62-8091-4f09-a0d9-41a0cdbd3673 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:46.075383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e793c271-7116-469b-bbdc-6686624610f4 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:46.104098 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4241fb-0de8-469c-9231-fae1acbfda0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:46.105123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4241fb-0de8-469c-9231-fae1acbfda0e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:46.105670 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4241fb-0de8-469c-9231-fae1acbfda0e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:46.106123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4241fb-0de8-469c-9231-fae1acbfda0e None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:46.106930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4241fb-0de8-469c-9231-fae1acbfda0e None admin] RBAC: Adding query filter params (name=compute) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:46.107547 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4241fb-0de8-469c-9231-fae1acbfda0e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:46.109231 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4241fb-0de8-469c-9231-fae1acbfda0e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:46.113411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1551b428-d164-4c93-96a2-a3a7a4c56fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:46.114042 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1551b428-d164-4c93-96a2-a3a7a4c56fd5 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:46.114319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1551b428-d164-4c93-96a2-a3a7a4c56fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:46.114590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1551b428-d164-4c93-96a2-a3a7a4c56fd5 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:46.115176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1551b428-d164-4c93-96a2-a3a7a4c56fd5 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:46.116544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1551b428-d164-4c93-96a2-a3a7a4c56fd5 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:46.116708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 492/995] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Sep 28 12:02:45 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:46.121875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4241fb-0de8-469c-9231-fae1acbfda0e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-1551b428-d164-4c93-96a2-a3a7a4c56fd5 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:46.128872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:46.129878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:46.131351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 504/996] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Sep 28 12:02:45 2022] GET /identity/v3/services/object-store => generated 93 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:46.136188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1551b428-d164-4c93-96a2-a3a7a4c56fd5 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:46.137358 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:46.237513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:46.243711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1551b428-d164-4c93-96a2-a3a7a4c56fd5 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:46.244105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4241fb-0de8-469c-9231-fae1acbfda0e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:46.281404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfd8cbc-6032-4627-87a7-6d5d127f29a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:46.282005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfd8cbc-6032-4627-87a7-6d5d127f29a2 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:46.282381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfd8cbc-6032-4627-87a7-6d5d127f29a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:46.282709 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfd8cbc-6032-4627-87a7-6d5d127f29a2 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:46.283390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfd8cbc-6032-4627-87a7-6d5d127f29a2 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:46.283697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfd8cbc-6032-4627-87a7-6d5d127f29a2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:46.285205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfd8cbc-6032-4627-87a7-6d5d127f29a2 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:46.286056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c8e2db-da07-43af-9e1c-c5cc3987eefb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:46.286640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c8e2db-da07-43af-9e1c-c5cc3987eefb None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:46.286929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c8e2db-da07-43af-9e1c-c5cc3987eefb None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:46.287210 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c8e2db-da07-43af-9e1c-c5cc3987eefb None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:46.287836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c8e2db-da07-43af-9e1c-c5cc3987eefb None admin] RBAC: Adding query filter params (type=compute) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:46.288145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c8e2db-da07-43af-9e1c-c5cc3987eefb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:46.289530 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c8e2db-da07-43af-9e1c-c5cc3987eefb None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:46.290861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 505/997] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Sep 28 12:02:46 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:46.296701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfd8cbc-6032-4627-87a7-6d5d127f29a2 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:46.303170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 493/998] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Sep 28 12:02:46 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:46.307685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c8e2db-da07-43af-9e1c-c5cc3987eefb None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:46.385606 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:46.411186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c8e2db-da07-43af-9e1c-c5cc3987eefb None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:46.418783 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfd8cbc-6032-4627-87a7-6d5d127f29a2 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:46.445697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acfcaa2-ebfa-44f6-9aab-3b02b881d8b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:46.446314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acfcaa2-ebfa-44f6-9aab-3b02b881d8b8 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:46.446598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acfcaa2-ebfa-44f6-9aab-3b02b881d8b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:46.447026 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acfcaa2-ebfa-44f6-9aab-3b02b881d8b8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:46.447724 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acfcaa2-ebfa-44f6-9aab-3b02b881d8b8 None admin] RBAC: Adding query filter params (interface=public, service_id=2be78e75fd6e4c108ed45c7ae8d97219, region_id=RegionOne) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:46.448086 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acfcaa2-ebfa-44f6-9aab-3b02b881d8b8 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:46.449515 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acfcaa2-ebfa-44f6-9aab-3b02b881d8b8 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:46.454798 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 494/999] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Sep 28 12:02:46 2022] GET /identity/v3/endpoints?service_id=2be78e75fd6e4c108ed45c7ae8d97219&interface=public®ion_id=RegionOne => generated 194 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:46.454930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaa7722-9870-46fc-9594-9c46ddb31784 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:46.455338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaa7722-9870-46fc-9594-9c46ddb31784 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:46.455733 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaa7722-9870-46fc-9594-9c46ddb31784 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:46.456014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaa7722-9870-46fc-9594-9c46ddb31784 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:46.456676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaa7722-9870-46fc-9594-9c46ddb31784 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:46.456987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaa7722-9870-46fc-9594-9c46ddb31784 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:46.458318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaa7722-9870-46fc-9594-9c46ddb31784 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:46.460317 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acfcaa2-ebfa-44f6-9aab-3b02b881d8b8 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:46.468370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 506/1000] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Sep 28 12:02:46 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:46.469336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaa7722-9870-46fc-9594-9c46ddb31784 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:46.506155 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:46.570585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acfcaa2-ebfa-44f6-9aab-3b02b881d8b8 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:46.587316 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaa7722-9870-46fc-9594-9c46ddb31784 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:46.615303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db55c6ce-d6f2-407f-86d3-a51a4fe2b9be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:46.616163 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db55c6ce-d6f2-407f-86d3-a51a4fe2b9be None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:46.616883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db55c6ce-d6f2-407f-86d3-a51a4fe2b9be None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:46.617165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db55c6ce-d6f2-407f-86d3-a51a4fe2b9be None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:46.617934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db55c6ce-d6f2-407f-86d3-a51a4fe2b9be None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:46.619305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db55c6ce-d6f2-407f-86d3-a51a4fe2b9be None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:46.624973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d845c3f-28e6-40ec-8f81-b9740e0e8044 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:46.633940 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 495/1001] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Sep 28 12:02:46 2022] GET /identity/v3/services => generated 1727 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:02:46.634167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d845c3f-28e6-40ec-8f81-b9740e0e8044 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:46.634167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d845c3f-28e6-40ec-8f81-b9740e0e8044 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:46.634167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d845c3f-28e6-40ec-8f81-b9740e0e8044 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:46.634167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d845c3f-28e6-40ec-8f81-b9740e0e8044 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:46.634167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d845c3f-28e6-40ec-8f81-b9740e0e8044 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:46.649052 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 507/1002] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:02:46 2022] POST /identity/v3/endpoints => generated 325 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:02:47.036249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af88892-dee1-4d30-8b6f-366f0a3c9921 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:47.036631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af88892-dee1-4d30-8b6f-366f0a3c9921 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:47.036836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af88892-dee1-4d30-8b6f-366f0a3c9921 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:47.038309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 496/1003] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:47.043220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f7b6b4-cdfd-4702-ae26-65e914b10432 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:47.043649 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f7b6b4-cdfd-4702-ae26-65e914b10432 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:47.043822 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f7b6b4-cdfd-4702-ae26-65e914b10432 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:47.062119 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-67f7b6b4-cdfd-4702-ae26-65e914b10432 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:47.063293 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-67f7b6b4-cdfd-4702-ae26-65e914b10432 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:47.137693 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 508/1004] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:47 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:47.402992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af88892-dee1-4d30-8b6f-366f0a3c9921 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:47.405889 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:47.511628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af88892-dee1-4d30-8b6f-366f0a3c9921 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:47.547965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6f083a-0720-492b-a218-1fa46d67157c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:47.548619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6f083a-0720-492b-a218-1fa46d67157c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:47.548896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6f083a-0720-492b-a218-1fa46d67157c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:47.549162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6f083a-0720-492b-a218-1fa46d67157c None admin] PATH_INFO: `/v3/services/image` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:47.549769 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6f083a-0720-492b-a218-1fa46d67157c None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:47.551130 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6f083a-0720-492b-a218-1fa46d67157c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-0d6f083a-0720-492b-a218-1fa46d67157c None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:47.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:47.561673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:47.562502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:47.562502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:47.562502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:47.562502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:47.562502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:47.562502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:47.562502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:47.562502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:47.562502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:47.562502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:47.562502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:47.562502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:47.562502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 28 12:02:47.562502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:47.564001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 497/1005] 10.222.0.26 () {56 vars in 1105 bytes} [Wed Sep 28 12:02:47 2022] GET /identity/v3/services/image => generated 86 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:47.567823 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f7b6b4-cdfd-4702-ae26-65e914b10432 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:47.579253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:47.666637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f7b6b4-cdfd-4702-ae26-65e914b10432 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:47.700022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e99442-aab0-44c6-a1cb-f007c34640cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:47.700637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e99442-aab0-44c6-a1cb-f007c34640cd None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:47.700914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e99442-aab0-44c6-a1cb-f007c34640cd None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:47.701182 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e99442-aab0-44c6-a1cb-f007c34640cd None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:47.701796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e99442-aab0-44c6-a1cb-f007c34640cd None admin] RBAC: Adding query filter params (name=image) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:47.702096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e99442-aab0-44c6-a1cb-f007c34640cd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:47.703427 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e99442-aab0-44c6-a1cb-f007c34640cd None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:47.708299 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 509/1006] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:47 2022] GET /identity/v3/services?name=image => generated 122 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:47.714212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6f083a-0720-492b-a218-1fa46d67157c None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:47.779245 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:47.815072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6f083a-0720-492b-a218-1fa46d67157c None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:47.851606 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b00aaed-6980-46d1-8b12-3250e9270a8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:47.852207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b00aaed-6980-46d1-8b12-3250e9270a8e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:47.852509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b00aaed-6980-46d1-8b12-3250e9270a8e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:47.852776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b00aaed-6980-46d1-8b12-3250e9270a8e None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:47.853401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b00aaed-6980-46d1-8b12-3250e9270a8e None admin] RBAC: Adding query filter params (type=image) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:47.853698 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b00aaed-6980-46d1-8b12-3250e9270a8e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:47.855045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b00aaed-6980-46d1-8b12-3250e9270a8e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:47.859714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 498/1007] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:47 2022] GET /identity/v3/services?type=image => generated 122 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:48.829966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9e6e34-27fd-4c17-b247-3e4064f6e2df None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:48.830489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9e6e34-27fd-4c17-b247-3e4064f6e2df None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:48.830489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9e6e34-27fd-4c17-b247-3e4064f6e2df None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:48.832583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 510/1008] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:48.837949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128c192a-bdf7-4653-8fb4-872483ebe015 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:48.838374 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128c192a-bdf7-4653-8fb4-872483ebe015 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:48.838516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128c192a-bdf7-4653-8fb4-872483ebe015 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:48.843357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf81a92-103e-4f2b-8d63-80ff7c0d0f17 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:48.843615 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf81a92-103e-4f2b-8d63-80ff7c0d0f17 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:48.843884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf81a92-103e-4f2b-8d63-80ff7c0d0f17 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:48.845283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 511/1009] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:48 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:48.849963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8690e514-91a0-4afb-803f-76c70ff5a340 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:48.850243 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8690e514-91a0-4afb-803f-76c70ff5a340 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:48.850515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8690e514-91a0-4afb-803f-76c70ff5a340 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:48.857593 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-128c192a-bdf7-4653-8fb4-872483ebe015 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:48.858796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-128c192a-bdf7-4653-8fb4-872483ebe015 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:48.868428 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-8690e514-91a0-4afb-803f-76c70ff5a340 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:48.869573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8690e514-91a0-4afb-803f-76c70ff5a340 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:48.927530 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 499/1010] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:48 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:48.940480 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 512/1011] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:48 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:49.197887 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128c192a-bdf7-4653-8fb4-872483ebe015 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:49.209655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8690e514-91a0-4afb-803f-76c70ff5a340 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:49.212956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:49.261908 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:49.304450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128c192a-bdf7-4653-8fb4-872483ebe015 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:49.322815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8690e514-91a0-4afb-803f-76c70ff5a340 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:49.343448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6d71ad-ac5e-4842-92f2-3e314736e0a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:49.344076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6d71ad-ac5e-4842-92f2-3e314736e0a0 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:49.344367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6d71ad-ac5e-4842-92f2-3e314736e0a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:49.344644 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6d71ad-ac5e-4842-92f2-3e314736e0a0 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:49.345223 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6d71ad-ac5e-4842-92f2-3e314736e0a0 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:49.347133 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6d71ad-ac5e-4842-92f2-3e314736e0a0 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-ed6d71ad-ac5e-4842-92f2-3e314736e0a0 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:49.355713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:49.356735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:49.357645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 500/1012] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:49 2022] GET /identity/v3/services/compute => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:49.362611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6d71ad-ac5e-4842-92f2-3e314736e0a0 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:49.363179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3d0fcd-280a-4f80-a282-a3690574b255 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:49.364011 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3d0fcd-280a-4f80-a282-a3690574b255 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:49.364523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3d0fcd-280a-4f80-a282-a3690574b255 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:49.364997 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3d0fcd-280a-4f80-a282-a3690574b255 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:49.367142 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3d0fcd-280a-4f80-a282-a3690574b255 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:49.368797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3d0fcd-280a-4f80-a282-a3690574b255 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:49.370827 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 513/1013] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:49.376345 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3d0fcd-280a-4f80-a282-a3690574b255 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:49.451033 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:49.470774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6d71ad-ac5e-4842-92f2-3e314736e0a0 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:49.486155 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3d0fcd-280a-4f80-a282-a3690574b255 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:49.499138 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:49.512429 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68234f50-76fe-46c4-9a37-0105d9aefb6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:49.513459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68234f50-76fe-46c4-9a37-0105d9aefb6f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:49.513633 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68234f50-76fe-46c4-9a37-0105d9aefb6f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:49.513903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68234f50-76fe-46c4-9a37-0105d9aefb6f None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:49.514532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68234f50-76fe-46c4-9a37-0105d9aefb6f None admin] RBAC: Adding query filter params (name=compute) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:49.514835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68234f50-76fe-46c4-9a37-0105d9aefb6f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:49.516209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68234f50-76fe-46c4-9a37-0105d9aefb6f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:49.523494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694e8390-1b69-46c3-a91b-d18baab89f85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:49.524102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694e8390-1b69-46c3-a91b-d18baab89f85 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:49.524410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694e8390-1b69-46c3-a91b-d18baab89f85 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:49.524687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694e8390-1b69-46c3-a91b-d18baab89f85 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:49.525332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694e8390-1b69-46c3-a91b-d18baab89f85 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:49.525822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 501/1014] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Sep 28 12:02:49 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:49.526658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68234f50-76fe-46c4-9a37-0105d9aefb6f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:49.526755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694e8390-1b69-46c3-a91b-d18baab89f85 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:49.555899 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 514/1015] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Sep 28 12:02:49 2022] POST /identity/v3/projects => generated 311 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:49.650761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68234f50-76fe-46c4-9a37-0105d9aefb6f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:49.689462 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b395ef48-4825-4a97-9b34-274b686e44d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:49.690127 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b395ef48-4825-4a97-9b34-274b686e44d1 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:49.690408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b395ef48-4825-4a97-9b34-274b686e44d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:49.690690 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b395ef48-4825-4a97-9b34-274b686e44d1 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:49.691345 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b395ef48-4825-4a97-9b34-274b686e44d1 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:49.691651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b395ef48-4825-4a97-9b34-274b686e44d1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:49.693092 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b395ef48-4825-4a97-9b34-274b686e44d1 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:49.703268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 502/1016] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Sep 28 12:02:49 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:49.704832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694e8390-1b69-46c3-a91b-d18baab89f85 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:49.813021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694e8390-1b69-46c3-a91b-d18baab89f85 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:49.837388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:49.851070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120aff79-5e1d-40ae-93c2-a4770a2bc495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:49.851671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120aff79-5e1d-40ae-93c2-a4770a2bc495 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:49.851945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120aff79-5e1d-40ae-93c2-a4770a2bc495 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:49.852214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120aff79-5e1d-40ae-93c2-a4770a2bc495 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:49.852916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120aff79-5e1d-40ae-93c2-a4770a2bc495 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:49.854270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120aff79-5e1d-40ae-93c2-a4770a2bc495 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:49.875527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 515/1017] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:02:49 2022] POST /identity/v3/endpoints => generated 334 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:50.050106 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92a89fe-5175-49b9-9ae2-82d2b9c9f380 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:50.050682 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92a89fe-5175-49b9-9ae2-82d2b9c9f380 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:50.050682 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92a89fe-5175-49b9-9ae2-82d2b9c9f380 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:50.052165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 503/1018] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:50.057861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffe75d3-5463-4809-88aa-d0a97be2d0f7 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:50.058155 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffe75d3-5463-4809-88aa-d0a97be2d0f7 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:50.058435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffe75d3-5463-4809-88aa-d0a97be2d0f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:50.077092 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-3ffe75d3-5463-4809-88aa-d0a97be2d0f7 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:50.078296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-3ffe75d3-5463-4809-88aa-d0a97be2d0f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:50.154714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 516/1019] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:50 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:50.435840 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92a89fe-5175-49b9-9ae2-82d2b9c9f380 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:50.493844 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:50.546548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92a89fe-5175-49b9-9ae2-82d2b9c9f380 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:50.583321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9aed0e-18cf-48b0-a354-651ebe9e65e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:50.583923 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9aed0e-18cf-48b0-a354-651ebe9e65e4 None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:50.584202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9aed0e-18cf-48b0-a354-651ebe9e65e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:50.584492 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9aed0e-18cf-48b0-a354-651ebe9e65e4 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:50.585137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9aed0e-18cf-48b0-a354-651ebe9e65e4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:50.586493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9aed0e-18cf-48b0-a354-651ebe9e65e4 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:50.604366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 504/1020] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Sep 28 12:02:50 2022] POST /identity/v3/services => generated 242 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:51.865123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad8619e-520b-4ada-a5c5-7320fbcd5056 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:51.865678 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad8619e-520b-4ada-a5c5-7320fbcd5056 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:51.865678 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad8619e-520b-4ada-a5c5-7320fbcd5056 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:51.867271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 517/1021] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:51.872539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cac050-ef89-4f08-9675-3ae761f9d485 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:51.872958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cac050-ef89-4f08-9675-3ae761f9d485 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:51.873122 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cac050-ef89-4f08-9675-3ae761f9d485 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:51.891557 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-38cac050-ef89-4f08-9675-3ae761f9d485 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:51.892768 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-38cac050-ef89-4f08-9675-3ae761f9d485 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:51.966483 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 505/1022] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:51 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:52.201685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fad607e-a546-431e-b332-c13f8fe9a25e None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:52.201895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fad607e-a546-431e-b332-c13f8fe9a25e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:52.202175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fad607e-a546-431e-b332-c13f8fe9a25e None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:52.203643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 518/1023] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:52.208878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f13a5f-ee27-4548-aec5-2d24dd264bcc None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:52.209169 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f13a5f-ee27-4548-aec5-2d24dd264bcc None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:52.209448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f13a5f-ee27-4548-aec5-2d24dd264bcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:52.227805 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-16f13a5f-ee27-4548-aec5-2d24dd264bcc None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:52.229019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-16f13a5f-ee27-4548-aec5-2d24dd264bcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:52.232617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fad607e-a546-431e-b332-c13f8fe9a25e None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:52.261175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:52.305268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 506/1024] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:52 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:52.341419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fad607e-a546-431e-b332-c13f8fe9a25e None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:52.349479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:52.378014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca35a18d-1cc3-4476-89e3-fe4c4b6c220a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:52.378631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca35a18d-1cc3-4476-89e3-fe4c4b6c220a None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:52.378903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca35a18d-1cc3-4476-89e3-fe4c4b6c220a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:52.379169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca35a18d-1cc3-4476-89e3-fe4c4b6c220a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:52.380268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca35a18d-1cc3-4476-89e3-fe4c4b6c220a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:52.381815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca35a18d-1cc3-4476-89e3-fe4c4b6c220a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:52.383521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 519/1025] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:52.389022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f13a5f-ee27-4548-aec5-2d24dd264bcc None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:52.500112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f13a5f-ee27-4548-aec5-2d24dd264bcc None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:52.534479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d26077-f449-4343-b921-7a8802aa026d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:52.535076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d26077-f449-4343-b921-7a8802aa026d None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:52.535349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d26077-f449-4343-b921-7a8802aa026d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:52.535616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d26077-f449-4343-b921-7a8802aa026d None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:52.536267 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d26077-f449-4343-b921-7a8802aa026d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:52.537717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d26077-f449-4343-b921-7a8802aa026d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:52.567533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 507/1026] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:02:52 2022] POST /identity/v3/users => generated 285 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:02:52.589360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca35a18d-1cc3-4476-89e3-fe4c4b6c220a None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:52.703120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca35a18d-1cc3-4476-89e3-fe4c4b6c220a None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:52.740253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4981f8ce-5c00-4e42-9a8f-a4b731c153b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:52.740934 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4981f8ce-5c00-4e42-9a8f-a4b731c153b1 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:52.741217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4981f8ce-5c00-4e42-9a8f-a4b731c153b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:52.741497 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4981f8ce-5c00-4e42-9a8f-a4b731c153b1 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:52.755835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4981f8ce-5c00-4e42-9a8f-a4b731c153b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-4981f8ce-5c00-4e42-9a8f-a4b731c153b1 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:52.760622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:52.761761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:52.762897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:52.762897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:52.762897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:52.762897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:52.762897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:52.762897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:52.762897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:52.762897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:52.762897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:52.762897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:52.762897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 28 12:02:52.762897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:52.762897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 520/1027] 10.222.0.26 () {56 vars in 1120 bytes} [Wed Sep 28 12:02:52 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:52.770341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d26077-f449-4343-b921-7a8802aa026d None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:52.794154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15830748-cd42-4e4f-9341-e746bfb88512 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:52.794435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15830748-cd42-4e4f-9341-e746bfb88512 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:52.794702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15830748-cd42-4e4f-9341-e746bfb88512 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:52.796198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 521/1028] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:52.802138 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d9190b-be31-4c2f-9316-bb2b033e6a27 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:52.802417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d9190b-be31-4c2f-9316-bb2b033e6a27 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:52.802997 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d9190b-be31-4c2f-9316-bb2b033e6a27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:52.805973 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:52.824673 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-71d9190b-be31-4c2f-9316-bb2b033e6a27 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:52.825875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-71d9190b-be31-4c2f-9316-bb2b033e6a27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:52.856605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:52.897429 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d26077-f449-4343-b921-7a8802aa026d None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:52.903536 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 522/1029] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:52 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:02:52.932026 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b67fc9e-57a7-4543-a829-eb4c90f2d259 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:52.932668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b67fc9e-57a7-4543-a829-eb4c90f2d259 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:52.932952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b67fc9e-57a7-4543-a829-eb4c90f2d259 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:52.933225 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b67fc9e-57a7-4543-a829-eb4c90f2d259 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:52.933888 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b67fc9e-57a7-4543-a829-eb4c90f2d259 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:52.934192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b67fc9e-57a7-4543-a829-eb4c90f2d259 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:52.935563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b67fc9e-57a7-4543-a829-eb4c90f2d259 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:52.948345 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 508/1030] 10.222.0.26 () {56 vars in 1115 bytes} [Wed Sep 28 12:02:52 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:52.949601 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d9190b-be31-4c2f-9316-bb2b033e6a27 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:53.049627 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d9190b-be31-4c2f-9316-bb2b033e6a27 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:53.083405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e183060-6f6c-455b-8dfd-676f08b4d6a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:53.084031 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e183060-6f6c-455b-8dfd-676f08b4d6a1 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:53.084332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e183060-6f6c-455b-8dfd-676f08b4d6a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:53.084610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e183060-6f6c-455b-8dfd-676f08b4d6a1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:53.094802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e183060-6f6c-455b-8dfd-676f08b4d6a1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:53.096255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e183060-6f6c-455b-8dfd-676f08b4d6a1 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-1e183060-6f6c-455b-8dfd-676f08b4d6a1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:53.113695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:53.114714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:53.121302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:53.121302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:53.121302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:53.121302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 523/1031] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:53.123416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b67fc9e-57a7-4543-a829-eb4c90f2d259 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:53.170025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e183060-6f6c-455b-8dfd-676f08b4d6a1 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:53.188496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:53.220045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:53.236477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b67fc9e-57a7-4543-a829-eb4c90f2d259 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:53.275740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70765bbb-f270-45cb-804e-d0eef28edd5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:53.276368 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70765bbb-f270-45cb-804e-d0eef28edd5a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:53.276490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e183060-6f6c-455b-8dfd-676f08b4d6a1 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:53.276664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70765bbb-f270-45cb-804e-d0eef28edd5a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:53.276920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70765bbb-f270-45cb-804e-d0eef28edd5a None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:53.277564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70765bbb-f270-45cb-804e-d0eef28edd5a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:53.278043 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70765bbb-f270-45cb-804e-d0eef28edd5a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:53.280190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70765bbb-f270-45cb-804e-d0eef28edd5a None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:53.299421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 509/1032] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:53.301065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70765bbb-f270-45cb-804e-d0eef28edd5a None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:53.309544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9cf710-c7f5-449c-be1e-bd9e1c7352d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:53.310169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9cf710-c7f5-449c-be1e-bd9e1c7352d7 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:53.310445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9cf710-c7f5-449c-be1e-bd9e1c7352d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:53.310719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9cf710-c7f5-449c-be1e-bd9e1c7352d7 None admin] PATH_INFO: `/v3/services/image` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:53.311296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9cf710-c7f5-449c-be1e-bd9e1c7352d7 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:53.312670 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9cf710-c7f5-449c-be1e-bd9e1c7352d7 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-da9cf710-c7f5-449c-be1e-bd9e1c7352d7 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:53.322666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:53.323669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:53.324540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 524/1033] 10.222.0.26 () {56 vars in 1105 bytes} [Wed Sep 28 12:02:53 2022] GET /identity/v3/services/image => generated 86 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:53.329407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9cf710-c7f5-449c-be1e-bd9e1c7352d7 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:53.412402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70765bbb-f270-45cb-804e-d0eef28edd5a None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:53.418536 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:53.440711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9cf710-c7f5-449c-be1e-bd9e1c7352d7 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:53.450918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54a7652-e16b-4b6a-8662-e5cb95f4ebd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:53.451565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54a7652-e16b-4b6a-8662-e5cb95f4ebd8 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:53.451856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54a7652-e16b-4b6a-8662-e5cb95f4ebd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:53.453975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54a7652-e16b-4b6a-8662-e5cb95f4ebd8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:53.463919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54a7652-e16b-4b6a-8662-e5cb95f4ebd8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:53.465966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54a7652-e16b-4b6a-8662-e5cb95f4ebd8 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-d54a7652-e16b-4b6a-8662-e5cb95f4ebd8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:53.479303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:53.480297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:53.486403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:53.486403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:53.486403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:53.486403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:53.486403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:53.486403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:53.486403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:53.486403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:53.486403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:02:53.486403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:53.486403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 510/1034] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Sep 28 12:02:53 2022] GET /identity/v3/users/nova => generated 82 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:53.486403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54a7652-e16b-4b6a-8662-e5cb95f4ebd8 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:53.495569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09769ff-ce14-4712-906a-2d93f3899028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:53.496049 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:53.496237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09769ff-ce14-4712-906a-2d93f3899028 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:53.496546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09769ff-ce14-4712-906a-2d93f3899028 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:53.496822 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09769ff-ce14-4712-906a-2d93f3899028 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:53.497461 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09769ff-ce14-4712-906a-2d93f3899028 None admin] RBAC: Adding query filter params (name=image) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:53.497765 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09769ff-ce14-4712-906a-2d93f3899028 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:53.499362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09769ff-ce14-4712-906a-2d93f3899028 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:53.505785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 525/1035] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:53 2022] GET /identity/v3/services?name=image => generated 122 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:53.511368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09769ff-ce14-4712-906a-2d93f3899028 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:53.600338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54a7652-e16b-4b6a-8662-e5cb95f4ebd8 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:53.623805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09769ff-ce14-4712-906a-2d93f3899028 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:53.633307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa9847c-599e-4194-ad24-7fde42b25242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:53.633970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa9847c-599e-4194-ad24-7fde42b25242 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:53.634249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa9847c-599e-4194-ad24-7fde42b25242 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:53.634517 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa9847c-599e-4194-ad24-7fde42b25242 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:53.635224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa9847c-599e-4194-ad24-7fde42b25242 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:53.635525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa9847c-599e-4194-ad24-7fde42b25242 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:53.636981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa9847c-599e-4194-ad24-7fde42b25242 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:53.653998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 511/1036] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Sep 28 12:02:53 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:53.660144 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa9847c-599e-4194-ad24-7fde42b25242 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:53.663575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5ba449-9e38-40a8-990a-c3c458ece4c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:53.664185 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5ba449-9e38-40a8-990a-c3c458ece4c9 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:53.664489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5ba449-9e38-40a8-990a-c3c458ece4c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:53.664760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5ba449-9e38-40a8-990a-c3c458ece4c9 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:53.665385 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5ba449-9e38-40a8-990a-c3c458ece4c9 None admin] RBAC: Adding query filter params (type=image) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:53.665682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5ba449-9e38-40a8-990a-c3c458ece4c9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:53.667016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5ba449-9e38-40a8-990a-c3c458ece4c9 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:53.672488 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 526/1037] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:53 2022] GET /identity/v3/services?type=image => generated 350 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:53.677888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5ba449-9e38-40a8-990a-c3c458ece4c9 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:53.688816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:53.731753 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:53.776535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa9847c-599e-4194-ad24-7fde42b25242 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:53.801421 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5ba449-9e38-40a8-990a-c3c458ece4c9 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:53.809503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2980663-d60e-4ad2-97b0-6a1ea2d30fe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:53.810116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2980663-d60e-4ad2-97b0-6a1ea2d30fe6 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:53.810395 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2980663-d60e-4ad2-97b0-6a1ea2d30fe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:53.810669 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2980663-d60e-4ad2-97b0-6a1ea2d30fe6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:53.840575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c352972f-f6c0-40ef-bf09-210969855911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:53.841433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c352972f-f6c0-40ef-bf09-210969855911 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:53.841914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c352972f-f6c0-40ef-bf09-210969855911 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:53.842253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c352972f-f6c0-40ef-bf09-210969855911 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:53.842942 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c352972f-f6c0-40ef-bf09-210969855911 None admin] RBAC: Adding query filter params (interface=public, service_id=cb6f112bcada453fafef4392cd3e7bf9, region_id=RegionOne) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:53.843244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c352972f-f6c0-40ef-bf09-210969855911 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:53.844596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c352972f-f6c0-40ef-bf09-210969855911 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:53.850235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 527/1038] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Sep 28 12:02:53 2022] GET /identity/v3/endpoints?service_id=cb6f112bcada453fafef4392cd3e7bf9&interface=public®ion_id=RegionOne => generated 194 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:53.855915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c352972f-f6c0-40ef-bf09-210969855911 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:53.881099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2980663-d60e-4ad2-97b0-6a1ea2d30fe6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:53.882693 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2980663-d60e-4ad2-97b0-6a1ea2d30fe6 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:53.925426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:53.951492 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:53.959745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c352972f-f6c0-40ef-bf09-210969855911 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:53.993330 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 512/1039] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:53 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 28 12:02:53.996096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f4946d-94c6-4070-bb38-1c352b3826f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:53.996861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f4946d-94c6-4070-bb38-1c352b3826f0 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:53.997142 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f4946d-94c6-4070-bb38-1c352b3826f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:53.997414 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f4946d-94c6-4070-bb38-1c352b3826f0 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:53.998116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f4946d-94c6-4070-bb38-1c352b3826f0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:53.998781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2980663-d60e-4ad2-97b0-6a1ea2d30fe6 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:53.999469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f4946d-94c6-4070-bb38-1c352b3826f0 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:54.008350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 528/1040] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Sep 28 12:02:53 2022] GET /identity/v3/services => generated 1957 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:02:54.118321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2980663-d60e-4ad2-97b0-6a1ea2d30fe6 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:54.155039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb6ee39-4460-4dbb-9d5c-df046d455da6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:54.155684 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb6ee39-4460-4dbb-9d5c-df046d455da6 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:54.155964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb6ee39-4460-4dbb-9d5c-df046d455da6 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:54.156235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb6ee39-4460-4dbb-9d5c-df046d455da6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:54.167676 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb6ee39-4460-4dbb-9d5c-df046d455da6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:54.169176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb6ee39-4460-4dbb-9d5c-df046d455da6 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-3bb6ee39-4460-4dbb-9d5c-df046d455da6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:54.184339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:54.185329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:54.186207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:54.186207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:54.186207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:54.186207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 513/1041] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:54.191362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f4946d-94c6-4070-bb38-1c352b3826f0 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:54.296423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f4946d-94c6-4070-bb38-1c352b3826f0 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:54.328707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3013b7-93e5-4412-a38c-738e315deb22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:54.329306 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3013b7-93e5-4412-a38c-738e315deb22 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:54.329581 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3013b7-93e5-4412-a38c-738e315deb22 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:54.329847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3013b7-93e5-4412-a38c-738e315deb22 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:54.331041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3013b7-93e5-4412-a38c-738e315deb22 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:54.331340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3013b7-93e5-4412-a38c-738e315deb22 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:54.332702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3013b7-93e5-4412-a38c-738e315deb22 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:54.335016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:54.351101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 529/1042] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:54.352802 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb6ee39-4460-4dbb-9d5c-df046d455da6 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:54.376697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:54.463082 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb6ee39-4460-4dbb-9d5c-df046d455da6 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:54.501569 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839a5ef4-7a49-4127-aa46-d85841d0d73e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:54.502200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839a5ef4-7a49-4127-aa46-d85841d0d73e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:54.503014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839a5ef4-7a49-4127-aa46-d85841d0d73e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:54.503285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839a5ef4-7a49-4127-aa46-d85841d0d73e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:54.510263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839a5ef4-7a49-4127-aa46-d85841d0d73e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:54.511700 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839a5ef4-7a49-4127-aa46-d85841d0d73e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-839a5ef4-7a49-4127-aa46-d85841d0d73e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:54.523303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:54.524179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:54.530150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:54.530150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:54.530150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:54.530150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:54.530150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:54.530150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:54.530150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:54.530150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:54.530150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 514/1043] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:54.532377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3013b7-93e5-4412-a38c-738e315deb22 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:54.634600 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3013b7-93e5-4412-a38c-738e315deb22 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:54.668926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dec7a2-970a-458b-8410-080703e98bcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:54.669532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dec7a2-970a-458b-8410-080703e98bcf None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:54.669813 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dec7a2-970a-458b-8410-080703e98bcf None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:54.670087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dec7a2-970a-458b-8410-080703e98bcf None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:54.670780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dec7a2-970a-458b-8410-080703e98bcf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:54.671086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dec7a2-970a-458b-8410-080703e98bcf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:54.672523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dec7a2-970a-458b-8410-080703e98bcf None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:54.684590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 530/1044] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:54.690677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839a5ef4-7a49-4127-aa46-d85841d0d73e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:54.774393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:54.798894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839a5ef4-7a49-4127-aa46-d85841d0d73e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:54.829808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7687f168-273e-4bbd-a5b7-0060ffe62e55 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:54.830210 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7687f168-273e-4bbd-a5b7-0060ffe62e55 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:54.830522 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7687f168-273e-4bbd-a5b7-0060ffe62e55 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:54.832195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 531/1045] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:54.833269 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bdcb60-2dc5-4692-a5e0-e38565c3ab39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:54.833872 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bdcb60-2dc5-4692-a5e0-e38565c3ab39 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:54.834153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bdcb60-2dc5-4692-a5e0-e38565c3ab39 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:54.834425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bdcb60-2dc5-4692-a5e0-e38565c3ab39 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:54.835148 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bdcb60-2dc5-4692-a5e0-e38565c3ab39 None admin] RBAC: Adding query filter params (role.id=d1813115b0cf4d2d905502119199b5db, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=8bf969348e7a4158acf8fb789430b0e6) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:54.835455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bdcb60-2dc5-4692-a5e0-e38565c3ab39 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:54.837001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bdcb60-2dc5-4692-a5e0-e38565c3ab39 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:54.837119 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3966f6d-b4c4-4cd6-b07c-1aad6edb3d3a None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:54.837288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3966f6d-b4c4-4cd6-b07c-1aad6edb3d3a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:54.837803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3966f6d-b4c4-4cd6-b07c-1aad6edb3d3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:54.848527 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 515/1046] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:54 2022] GET /identity/v3/role_assignments?user.id=8bf969348e7a4158acf8fb789430b0e6&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=d1813115b0cf4d2d905502119199b5db => generated 259 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:54.862339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-c3966f6d-b4c4-4cd6-b07c-1aad6edb3d3a None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:54.863684 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c3966f6d-b4c4-4cd6-b07c-1aad6edb3d3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:54.921837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:54.944673 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 532/1047] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:54 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:55.208057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bdcb60-2dc5-4692-a5e0-e38565c3ab39 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:55.309654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bdcb60-2dc5-4692-a5e0-e38565c3ab39 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:55.344108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca82c1c9-863a-4c34-be6f-7f7e7e4235c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:55.344767 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca82c1c9-863a-4c34-be6f-7f7e7e4235c3 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:55.345046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca82c1c9-863a-4c34-be6f-7f7e7e4235c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:55.345318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca82c1c9-863a-4c34-be6f-7f7e7e4235c3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:55.352869 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:55.356760 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca82c1c9-863a-4c34-be6f-7f7e7e4235c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-ca82c1c9-863a-4c34-be6f-7f7e7e4235c3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:55.362110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:55.363246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:55.364301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:55.364301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:55.364301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:55.364301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:55.364301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:55.364301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:55.364301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:55.364301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:55.364301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:55.364301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:55.364301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:55.364301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:55.366804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 516/1048] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:02:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:55.372540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3966f6d-b4c4-4cd6-b07c-1aad6edb3d3a None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:55.469910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3966f6d-b4c4-4cd6-b07c-1aad6edb3d3a None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:55.511900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3f4237-5045-457b-9ee6-9912c96fff41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:55.512540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3f4237-5045-457b-9ee6-9912c96fff41 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:55.512848 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3f4237-5045-457b-9ee6-9912c96fff41 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:55.513087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3f4237-5045-457b-9ee6-9912c96fff41 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:55.513749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3f4237-5045-457b-9ee6-9912c96fff41 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:55.514048 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3f4237-5045-457b-9ee6-9912c96fff41 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:55.515389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3f4237-5045-457b-9ee6-9912c96fff41 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:55.529022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 533/1049] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:02:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:55.530292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca82c1c9-863a-4c34-be6f-7f7e7e4235c3 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:55.634845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca82c1c9-863a-4c34-be6f-7f7e7e4235c3 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:55.668747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f13ec29-6caf-4584-bf35-9dd29beeb496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:55.669375 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f13ec29-6caf-4584-bf35-9dd29beeb496 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:55.669658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f13ec29-6caf-4584-bf35-9dd29beeb496 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:55.669932 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f13ec29-6caf-4584-bf35-9dd29beeb496 None admin] PATH_INFO: `/v3/users/89df2cea6cb14b4db32b364990dacbf1` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:55.684894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f13ec29-6caf-4584-bf35-9dd29beeb496 None admin] RBAC: Authorizing `identity:get_user(user_id=89df2cea6cb14b4db32b364990dacbf1)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:55.686346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f13ec29-6caf-4584-bf35-9dd29beeb496 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:55.693201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 517/1050] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:02:55 2022] GET /identity/v3/users/89df2cea6cb14b4db32b364990dacbf1 => generated 285 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:55.694359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3f4237-5045-457b-9ee6-9912c96fff41 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:55.746091 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:55.796018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3f4237-5045-457b-9ee6-9912c96fff41 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:55.828661 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2528c98-396a-409e-b9ea-f800bb6318cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:55.829264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2528c98-396a-409e-b9ea-f800bb6318cf None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:55.829548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2528c98-396a-409e-b9ea-f800bb6318cf None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:55.829819 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2528c98-396a-409e-b9ea-f800bb6318cf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:55.895757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2528c98-396a-409e-b9ea-f800bb6318cf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:55.897235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2528c98-396a-409e-b9ea-f800bb6318cf None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:55.944289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:56.001726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 534/1051] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:55 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:02:56.003421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f13ec29-6caf-4584-bf35-9dd29beeb496 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:56.062226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:56.108456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f13ec29-6caf-4584-bf35-9dd29beeb496 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:56.144965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d662c56-90ae-4c77-a11a-41a14ab46b2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:56.145592 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d662c56-90ae-4c77-a11a-41a14ab46b2d None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:56.145869 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d662c56-90ae-4c77-a11a-41a14ab46b2d None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:56.146139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d662c56-90ae-4c77-a11a-41a14ab46b2d None admin] PATH_INFO: `/v3/projects/28a0c86e6c374712b465b9b45443db89` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:56.147265 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d662c56-90ae-4c77-a11a-41a14ab46b2d None admin] RBAC: Authorizing `identity:get_project(project_id=28a0c86e6c374712b465b9b45443db89)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:56.148718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d662c56-90ae-4c77-a11a-41a14ab46b2d None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:56.150499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 518/1052] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:02:56 2022] GET /identity/v3/projects/28a0c86e6c374712b465b9b45443db89 => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:56.156371 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2528c98-396a-409e-b9ea-f800bb6318cf None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:56.210172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7010a126-bedd-4c58-be74-97770b966320 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:56.210724 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7010a126-bedd-4c58-be74-97770b966320 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:56.210868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7010a126-bedd-4c58-be74-97770b966320 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:56.212236 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 519/1053] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:56.217482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d29c2f3-cf28-4e81-9ea0-a00165c8ebfd None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:56.217768 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d29c2f3-cf28-4e81-9ea0-a00165c8ebfd None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:56.218048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d29c2f3-cf28-4e81-9ea0-a00165c8ebfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:56.236616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-5d29c2f3-cf28-4e81-9ea0-a00165c8ebfd None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:56.238371 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5d29c2f3-cf28-4e81-9ea0-a00165c8ebfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:56.256843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2528c98-396a-409e-b9ea-f800bb6318cf None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:56.295949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48268ac9-1f2d-49b7-b508-d951d2982fdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:56.297110 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48268ac9-1f2d-49b7-b508-d951d2982fdb None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:56.297982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48268ac9-1f2d-49b7-b508-d951d2982fdb None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:56.298442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48268ac9-1f2d-49b7-b508-d951d2982fdb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:56.300141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48268ac9-1f2d-49b7-b508-d951d2982fdb None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=28a0c86e6c374712b465b9b45443db89, user.id=89df2cea6cb14b4db32b364990dacbf1) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:56.300762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48268ac9-1f2d-49b7-b508-d951d2982fdb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:56.302493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48268ac9-1f2d-49b7-b508-d951d2982fdb None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:56.306112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 520/1054] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:56 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:56.316347 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 535/1055] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:02:56 2022] GET /identity/v3/role_assignments?user.id=89df2cea6cb14b4db32b364990dacbf1&scope.project.id=28a0c86e6c374712b465b9b45443db89&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 259 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:56.579915 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d29c2f3-cf28-4e81-9ea0-a00165c8ebfd None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:56.639086 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:56.684625 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d29c2f3-cf28-4e81-9ea0-a00165c8ebfd None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:56.720352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46e15e5-ee26-4099-9791-8d66a85c580a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:56.721018 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46e15e5-ee26-4099-9791-8d66a85c580a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:56.721821 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46e15e5-ee26-4099-9791-8d66a85c580a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:56.722095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46e15e5-ee26-4099-9791-8d66a85c580a None admin] PATH_INFO: `/v3/services/image` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:56.722690 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46e15e5-ee26-4099-9791-8d66a85c580a None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:56.724063 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46e15e5-ee26-4099-9791-8d66a85c580a None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-e46e15e5-ee26-4099-9791-8d66a85c580a None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:56.733025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:56.733991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:56.734841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 521/1056] 10.222.0.26 () {56 vars in 1105 bytes} [Wed Sep 28 12:02:56 2022] GET /identity/v3/services/image => generated 86 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:56.741862 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48268ac9-1f2d-49b7-b508-d951d2982fdb None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:56.757625 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:56.845960 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48268ac9-1f2d-49b7-b508-d951d2982fdb None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:56.881174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a237f829-8bd8-4f0d-876e-4bbb5e6f7192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:56.881863 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a237f829-8bd8-4f0d-876e-4bbb5e6f7192 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:56.882146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a237f829-8bd8-4f0d-876e-4bbb5e6f7192 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:56.882422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a237f829-8bd8-4f0d-876e-4bbb5e6f7192 None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:56.883129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a237f829-8bd8-4f0d-876e-4bbb5e6f7192 None admin] RBAC: Adding query filter params (name=image) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:56.883440 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a237f829-8bd8-4f0d-876e-4bbb5e6f7192 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:56.884887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a237f829-8bd8-4f0d-876e-4bbb5e6f7192 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:56.890632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 536/1057] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:56 2022] GET /identity/v3/services?name=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:56.897010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46e15e5-ee26-4099-9791-8d66a85c580a None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:56.988418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9760ac56-03ae-4a47-8c1c-c593f69c5dff None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:56.988660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9760ac56-03ae-4a47-8c1c-c593f69c5dff None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:56.988935 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9760ac56-03ae-4a47-8c1c-c593f69c5dff None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:56.990266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 537/1058] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:56.995185 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f8ccdd-666d-42e9-a4f5-903bcc2fc6fc None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:56.995478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f8ccdd-666d-42e9-a4f5-903bcc2fc6fc None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:56.995762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f8ccdd-666d-42e9-a4f5-903bcc2fc6fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:56.999340 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46e15e5-ee26-4099-9791-8d66a85c580a None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:57.014526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-23f8ccdd-666d-42e9-a4f5-903bcc2fc6fc None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:57.015708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-23f8ccdd-666d-42e9-a4f5-903bcc2fc6fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:57.020278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:57.036346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c43c804-00cd-4246-a3b9-0e766ff315a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:57.036989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c43c804-00cd-4246-a3b9-0e766ff315a0 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:57.037274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c43c804-00cd-4246-a3b9-0e766ff315a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:57.037551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c43c804-00cd-4246-a3b9-0e766ff315a0 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:57.038185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c43c804-00cd-4246-a3b9-0e766ff315a0 None admin] RBAC: Adding query filter params (type=image) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:57.038489 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c43c804-00cd-4246-a3b9-0e766ff315a0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:57.039864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c43c804-00cd-4246-a3b9-0e766ff315a0 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:57.044800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 522/1059] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:56 2022] GET /identity/v3/services?type=image => generated 350 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:57.050901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c43c804-00cd-4246-a3b9-0e766ff315a0 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:57.085027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 538/1060] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:56 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:57.148734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c43c804-00cd-4246-a3b9-0e766ff315a0 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:57.183415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402fead0-1bb6-4117-ad56-1d0c449acb11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:57.184032 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402fead0-1bb6-4117-ad56-1d0c449acb11 None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:57.184329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402fead0-1bb6-4117-ad56-1d0c449acb11 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:57.184614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402fead0-1bb6-4117-ad56-1d0c449acb11 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:57.185305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402fead0-1bb6-4117-ad56-1d0c449acb11 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:57.186680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402fead0-1bb6-4117-ad56-1d0c449acb11 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:57.207766 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 523/1061] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:02:57 2022] POST /identity/v3/endpoints => generated 327 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:02:57.354450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f8ccdd-666d-42e9-a4f5-903bcc2fc6fc None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:57.364021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:57.461515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f8ccdd-666d-42e9-a4f5-903bcc2fc6fc None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:57.509379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e458733-8ff5-4428-ae42-37722d4dac28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:57.510025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e458733-8ff5-4428-ae42-37722d4dac28 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:57.510308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e458733-8ff5-4428-ae42-37722d4dac28 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:57.510580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e458733-8ff5-4428-ae42-37722d4dac28 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:57.520883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e458733-8ff5-4428-ae42-37722d4dac28 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-7e458733-8ff5-4428-ae42-37722d4dac28 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:57.525652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:57.526573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:57.527426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:57.527426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:57.527426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:57.527426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:57.527426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:57.527426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:57.527426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:57.527426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:57.527426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:57.527426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:57.527426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 28 12:02:57.527426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:57.527426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 539/1062] 10.222.0.26 () {56 vars in 1120 bytes} [Wed Sep 28 12:02:57 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:57.534706 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402fead0-1bb6-4117-ad56-1d0c449acb11 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:57.577174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:57.640746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402fead0-1bb6-4117-ad56-1d0c449acb11 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:57.675317 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1d0f92-d9be-4089-a46e-1af0844c464b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:57.675925 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1d0f92-d9be-4089-a46e-1af0844c464b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:57.676206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1d0f92-d9be-4089-a46e-1af0844c464b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:57.676510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1d0f92-d9be-4089-a46e-1af0844c464b None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:57.677179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1d0f92-d9be-4089-a46e-1af0844c464b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:57.677482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1d0f92-d9be-4089-a46e-1af0844c464b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:57.678853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1d0f92-d9be-4089-a46e-1af0844c464b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:57.691801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 524/1063] 10.222.0.26 () {56 vars in 1115 bytes} [Wed Sep 28 12:02:57 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:57.693021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e458733-8ff5-4428-ae42-37722d4dac28 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:57.755459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:57.794587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e458733-8ff5-4428-ae42-37722d4dac28 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:57.827365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06312f01-358f-4233-95a6-793dd699f946 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:57.827986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06312f01-358f-4233-95a6-793dd699f946 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:57.828263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06312f01-358f-4233-95a6-793dd699f946 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:57.828555 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06312f01-358f-4233-95a6-793dd699f946 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:57.838448 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06312f01-358f-4233-95a6-793dd699f946 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:57.839877 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06312f01-358f-4233-95a6-793dd699f946 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-06312f01-358f-4233-95a6-793dd699f946 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:57.854302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:57.855268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:57.861014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:57.861014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:57.861014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:57.861014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 540/1064] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:57.861680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1d0f92-d9be-4089-a46e-1af0844c464b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:57.964394 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1d0f92-d9be-4089-a46e-1af0844c464b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:57.965975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:57.999975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd9afae-29f2-4305-8167-c6fc72f270d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:58.000615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd9afae-29f2-4305-8167-c6fc72f270d2 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:58.000898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd9afae-29f2-4305-8167-c6fc72f270d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:58.001169 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd9afae-29f2-4305-8167-c6fc72f270d2 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:58.001808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd9afae-29f2-4305-8167-c6fc72f270d2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:58.002110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd9afae-29f2-4305-8167-c6fc72f270d2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:58.003712 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd9afae-29f2-4305-8167-c6fc72f270d2 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:58.015675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 525/1065] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:58.021666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06312f01-358f-4233-95a6-793dd699f946 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:58.122238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06312f01-358f-4233-95a6-793dd699f946 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:58.137589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:58.158183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fcdae8-3f5f-4b38-9379-33a40cc70369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:58.158832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fcdae8-3f5f-4b38-9379-33a40cc70369 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:58.159110 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fcdae8-3f5f-4b38-9379-33a40cc70369 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:58.159381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fcdae8-3f5f-4b38-9379-33a40cc70369 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:58.166765 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fcdae8-3f5f-4b38-9379-33a40cc70369 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:58.168179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fcdae8-3f5f-4b38-9379-33a40cc70369 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-62fcdae8-3f5f-4b38-9379-33a40cc70369 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:02:58.179546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:58.180508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:58.181460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:58.181460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:58.181460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:02:58.181460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:02:58.181460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:02:58.181460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:02:58.181460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:02:58.181460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:02:58.181460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:02:58.181460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:58.184613 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 541/1066] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Sep 28 12:02:58 2022] GET /identity/v3/users/nova => generated 82 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:02:58.186287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd9afae-29f2-4305-8167-c6fc72f270d2 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:58.284574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd9afae-29f2-4305-8167-c6fc72f270d2 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:58.317788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aeff292-7865-41c5-9073-0e02d9d10ed9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:58.318387 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aeff292-7865-41c5-9073-0e02d9d10ed9 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:58.318667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aeff292-7865-41c5-9073-0e02d9d10ed9 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:58.318941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aeff292-7865-41c5-9073-0e02d9d10ed9 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:58.319664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aeff292-7865-41c5-9073-0e02d9d10ed9 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:58.319970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aeff292-7865-41c5-9073-0e02d9d10ed9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:58.321421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aeff292-7865-41c5-9073-0e02d9d10ed9 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:58.343270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 526/1067] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Sep 28 12:02:58 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:58.344580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fcdae8-3f5f-4b38-9379-33a40cc70369 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:58.452023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fcdae8-3f5f-4b38-9379-33a40cc70369 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:58.492631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4282b1a8-229e-4d56-a089-7e216be61017 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:58.492950 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4282b1a8-229e-4d56-a089-7e216be61017 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:58.493188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623fe1fb-4e68-47c3-8273-3dc2f3b07498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:58.493315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4282b1a8-229e-4d56-a089-7e216be61017 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:58.493823 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623fe1fb-4e68-47c3-8273-3dc2f3b07498 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:58.494097 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623fe1fb-4e68-47c3-8273-3dc2f3b07498 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:58.494367 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623fe1fb-4e68-47c3-8273-3dc2f3b07498 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:58.494798 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 527/1068] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:02:58.499694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dd5e4e-bfed-4bac-bd9e-17fd96d2eb9d None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:58.499981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dd5e4e-bfed-4bac-bd9e-17fd96d2eb9d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:58.500259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dd5e4e-bfed-4bac-bd9e-17fd96d2eb9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:58.505486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623fe1fb-4e68-47c3-8273-3dc2f3b07498 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:58.507176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623fe1fb-4e68-47c3-8273-3dc2f3b07498 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:58.517975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-09dd5e4e-bfed-4bac-bd9e-17fd96d2eb9d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:58.519256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-09dd5e4e-bfed-4bac-bd9e-17fd96d2eb9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-623fe1fb-4e68-47c3-8273-3dc2f3b07498 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:02:58.523820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:58.524884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:02:58.525807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:02:58.525807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:02:58.525807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:02:58.528482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 542/1069] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:02:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:58.530382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623fe1fb-4e68-47c3-8273-3dc2f3b07498 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:58.536572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:58.560912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:58.591532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 528/1070] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:58.636592 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623fe1fb-4e68-47c3-8273-3dc2f3b07498 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:58.670579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27503791-2440-4035-a84f-6d3c1b359dab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:58.671177 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27503791-2440-4035-a84f-6d3c1b359dab None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:58.671456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27503791-2440-4035-a84f-6d3c1b359dab None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:58.671727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27503791-2440-4035-a84f-6d3c1b359dab None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:58.672387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27503791-2440-4035-a84f-6d3c1b359dab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:58.672690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27503791-2440-4035-a84f-6d3c1b359dab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:58.674035 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27503791-2440-4035-a84f-6d3c1b359dab None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:58.690367 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 543/1071] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:02:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:02:58.691273 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09dd5e4e-bfed-4bac-bd9e-17fd96d2eb9d None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:58.788262 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09dd5e4e-bfed-4bac-bd9e-17fd96d2eb9d None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:58.825836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c30d66-e75a-426b-aa91-4cd4d913598f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:58.826466 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c30d66-e75a-426b-aa91-4cd4d913598f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:58.826741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c30d66-e75a-426b-aa91-4cd4d913598f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:58.827010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c30d66-e75a-426b-aa91-4cd4d913598f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:58.834360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c30d66-e75a-426b-aa91-4cd4d913598f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:58.835801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c30d66-e75a-426b-aa91-4cd4d913598f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-13c30d66-e75a-426b-aa91-4cd4d913598f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:02:58.847809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:58.848717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:58.854418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:02:58.854418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:02:58.854418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:02:58.854418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:02:58.854418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:02:58.854418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:02:58.854418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:02:58.854418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:58.854418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 529/1072] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:02:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:58.854919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27503791-2440-4035-a84f-6d3c1b359dab None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:58.864172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c30d66-e75a-426b-aa91-4cd4d913598f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:58.878456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:58.952514 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:58.970860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27503791-2440-4035-a84f-6d3c1b359dab None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:58.978885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c30d66-e75a-426b-aa91-4cd4d913598f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:59.009584 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fc22a4-8ce3-430c-b163-68fa542c5277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:59.010192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fc22a4-8ce3-430c-b163-68fa542c5277 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:59.010467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fc22a4-8ce3-430c-b163-68fa542c5277 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:59.010981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fc22a4-8ce3-430c-b163-68fa542c5277 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:59.011487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fc22a4-8ce3-430c-b163-68fa542c5277 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:59.011853 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fc22a4-8ce3-430c-b163-68fa542c5277 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:59.013298 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fc22a4-8ce3-430c-b163-68fa542c5277 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:59.016167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ba118a-c201-42c8-a2f3-110b266c8c75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:59.016832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ba118a-c201-42c8-a2f3-110b266c8c75 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:59.017114 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ba118a-c201-42c8-a2f3-110b266c8c75 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:59.017394 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ba118a-c201-42c8-a2f3-110b266c8c75 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:59.030096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 544/1073] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:02:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:59.031143 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fc22a4-8ce3-430c-b163-68fa542c5277 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:59.036202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ba118a-c201-42c8-a2f3-110b266c8c75 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-98ba118a-c201-42c8-a2f3-110b266c8c75 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:02:59.041090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:02:59.042195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:02:59.043207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:02:59.043207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:02:59.043207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:02:59.043207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:02:59.043207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:02:59.043207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:02:59.043207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:02:59.043207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:02:59.043207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:02:59.043207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:02:59.043207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:02:59.043207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:02:59.048383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 530/1074] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:02:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:02:59.050037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ba118a-c201-42c8-a2f3-110b266c8c75 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:59.140037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:59.141462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fc22a4-8ce3-430c-b163-68fa542c5277 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:59.173023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ba118a-c201-42c8-a2f3-110b266c8c75 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:59.181451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e19872-21c5-4ae3-90b1-1af602a09484 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:59.182108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e19872-21c5-4ae3-90b1-1af602a09484 None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:59.182383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e19872-21c5-4ae3-90b1-1af602a09484 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:59.182657 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e19872-21c5-4ae3-90b1-1af602a09484 None admin] PATH_INFO: `/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/8bf969348e7a4158acf8fb789430b0e6/roles/d1813115b0cf4d2d905502119199b5db` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:59.185180 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e19872-21c5-4ae3-90b1-1af602a09484 None admin] RBAC: Authorizing `identity:create_grant(project_id=8695d562e3e24c7c9300cf3a23cd7502, user_id=8bf969348e7a4158acf8fb789430b0e6, role_id=d1813115b0cf4d2d905502119199b5db)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:59.186592 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e19872-21c5-4ae3-90b1-1af602a09484 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:59.201453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 545/1075] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:02:59 2022] PUT /identity/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/8bf969348e7a4158acf8fb789430b0e6/roles/d1813115b0cf4d2d905502119199b5db => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:02:59.209743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbd6b50-525b-4a60-ba94-d76c1c5f0ae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:59.210562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbd6b50-525b-4a60-ba94-d76c1c5f0ae6 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:59.210943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbd6b50-525b-4a60-ba94-d76c1c5f0ae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:59.211319 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbd6b50-525b-4a60-ba94-d76c1c5f0ae6 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:59.212115 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbd6b50-525b-4a60-ba94-d76c1c5f0ae6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:02:59.212556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbd6b50-525b-4a60-ba94-d76c1c5f0ae6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:59.214364 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbd6b50-525b-4a60-ba94-d76c1c5f0ae6 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:59.230075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 531/1076] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:02:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:59.231516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e19872-21c5-4ae3-90b1-1af602a09484 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:59.243068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:59.348554 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e19872-21c5-4ae3-90b1-1af602a09484 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:59.390163 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd266a2-430f-4ff0-af50-2812d2bc03be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:59.390806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd266a2-430f-4ff0-af50-2812d2bc03be None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:59.391175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd266a2-430f-4ff0-af50-2812d2bc03be None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:59.391695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd266a2-430f-4ff0-af50-2812d2bc03be None admin] PATH_INFO: `/v3/users/89df2cea6cb14b4db32b364990dacbf1` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:59.393907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd266a2-430f-4ff0-af50-2812d2bc03be None admin] RBAC: Authorizing `identity:get_user(user_id=89df2cea6cb14b4db32b364990dacbf1)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:59.395324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd266a2-430f-4ff0-af50-2812d2bc03be None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:59.397119 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 546/1077] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:02:59 2022] GET /identity/v3/users/89df2cea6cb14b4db32b364990dacbf1 => generated 285 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:59.403033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbd6b50-525b-4a60-ba94-d76c1c5f0ae6 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:59.465305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c2befb-49ae-4f93-91ee-5915af75a3d3 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:59.465586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c2befb-49ae-4f93-91ee-5915af75a3d3 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:59.465870 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c2befb-49ae-4f93-91ee-5915af75a3d3 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:59.467491 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 547/1078] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:02:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:02:59.473626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5756f525-76e1-4e0d-a458-a2585dc5b8d1 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:59.474015 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5756f525-76e1-4e0d-a458-a2585dc5b8d1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:59.474409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5756f525-76e1-4e0d-a458-a2585dc5b8d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:59.496433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-5756f525-76e1-4e0d-a458-a2585dc5b8d1 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:02:59.497745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5756f525-76e1-4e0d-a458-a2585dc5b8d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:02:59.515854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbd6b50-525b-4a60-ba94-d76c1c5f0ae6 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:59.548366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:59.555342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc575cd-48f5-4f2d-aa47-fd6764ebd105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:59.556383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc575cd-48f5-4f2d-aa47-fd6764ebd105 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:59.556735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc575cd-48f5-4f2d-aa47-fd6764ebd105 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:59.557081 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc575cd-48f5-4f2d-aa47-fd6764ebd105 None admin] PATH_INFO: `/v3/projects/28a0c86e6c374712b465b9b45443db89` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:59.558750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc575cd-48f5-4f2d-aa47-fd6764ebd105 None admin] RBAC: Authorizing `identity:get_project(project_id=28a0c86e6c374712b465b9b45443db89)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:59.560278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc575cd-48f5-4f2d-aa47-fd6764ebd105 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:59.562213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 532/1079] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:02:59 2022] GET /identity/v3/projects/28a0c86e6c374712b465b9b45443db89 => generated 311 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:02:59.567845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc575cd-48f5-4f2d-aa47-fd6764ebd105 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:59.577497 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 548/1080] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:02:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:02:59.671672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc575cd-48f5-4f2d-aa47-fd6764ebd105 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:59.706014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9356048d-ba89-4316-86ad-d2abd0453497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:59.706666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9356048d-ba89-4316-86ad-d2abd0453497 None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:59.706945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9356048d-ba89-4316-86ad-d2abd0453497 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:59.707215 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9356048d-ba89-4316-86ad-d2abd0453497 None admin] PATH_INFO: `/v3/projects/28a0c86e6c374712b465b9b45443db89/users/89df2cea6cb14b4db32b364990dacbf1/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:02:59.709202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9356048d-ba89-4316-86ad-d2abd0453497 None admin] RBAC: Authorizing `identity:create_grant(project_id=28a0c86e6c374712b465b9b45443db89, user_id=89df2cea6cb14b4db32b364990dacbf1, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:02:59.712043 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9356048d-ba89-4316-86ad-d2abd0453497 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:02:59.725162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 533/1081] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:02:59 2022] PUT /identity/v3/projects/28a0c86e6c374712b465b9b45443db89/users/89df2cea6cb14b4db32b364990dacbf1/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:02:59.846433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5756f525-76e1-4e0d-a458-a2585dc5b8d1 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:02:59.849712 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:02:59.951288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5756f525-76e1-4e0d-a458-a2585dc5b8d1 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:02:59.990385 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527d9f0e-d8e7-4572-96a8-daca01a4092c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:02:59.991288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527d9f0e-d8e7-4572-96a8-daca01a4092c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:02:59.991757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527d9f0e-d8e7-4572-96a8-daca01a4092c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:02:59.992189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527d9f0e-d8e7-4572-96a8-daca01a4092c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:00.060125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527d9f0e-d8e7-4572-96a8-daca01a4092c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:00.062254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527d9f0e-d8e7-4572-96a8-daca01a4092c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:00.065601 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:00.168356 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 549/1082] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:02:59 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:03:00.173199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9356048d-ba89-4316-86ad-d2abd0453497 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:00.234549 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:00.278428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9356048d-ba89-4316-86ad-d2abd0453497 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:00.313146 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e66f2e0-3943-45f1-8fdb-df293b5f0f3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:00.313775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e66f2e0-3943-45f1-8fdb-df293b5f0f3c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:00.314056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e66f2e0-3943-45f1-8fdb-df293b5f0f3c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:00.314332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e66f2e0-3943-45f1-8fdb-df293b5f0f3c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:00.325097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e66f2e0-3943-45f1-8fdb-df293b5f0f3c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:00.326548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e66f2e0-3943-45f1-8fdb-df293b5f0f3c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-9e66f2e0-3943-45f1-8fdb-df293b5f0f3c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:03:00.342172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:03:00.343078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:03:00.343966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:03:00.343966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:03:00.343966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:03:00.343966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 534/1083] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:03:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:03:00.350290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527d9f0e-d8e7-4572-96a8-daca01a4092c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:00.451073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527d9f0e-d8e7-4572-96a8-daca01a4092c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:00.453499 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:00.499175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3122f45-57fb-4d8b-969f-deeb9046fdf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:00.500925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3122f45-57fb-4d8b-969f-deeb9046fdf0 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:00.501229 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3122f45-57fb-4d8b-969f-deeb9046fdf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:00.501492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3122f45-57fb-4d8b-969f-deeb9046fdf0 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:00.503173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3122f45-57fb-4d8b-969f-deeb9046fdf0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:00.503484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3122f45-57fb-4d8b-969f-deeb9046fdf0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:00.504888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3122f45-57fb-4d8b-969f-deeb9046fdf0 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:00.520180 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 550/1084] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:03:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:03:01.394098 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2513eaea-b8ba-4e19-ae5f-4f63d3a5f73d None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:01.394628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2513eaea-b8ba-4e19-ae5f-4f63d3a5f73d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:01.394628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2513eaea-b8ba-4e19-ae5f-4f63d3a5f73d None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:01.395982 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 535/1085] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:01.401447 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436a2056-b4b3-4e8f-bccf-fe559cea2ee5 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:01.401854 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436a2056-b4b3-4e8f-bccf-fe559cea2ee5 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:01.402108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436a2056-b4b3-4e8f-bccf-fe559cea2ee5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:01.420222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-436a2056-b4b3-4e8f-bccf-fe559cea2ee5 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:01.421404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-436a2056-b4b3-4e8f-bccf-fe559cea2ee5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:01.485327 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 551/1086] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:01.751272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2513eaea-b8ba-4e19-ae5f-4f63d3a5f73d None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:01.855366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2513eaea-b8ba-4e19-ae5f-4f63d3a5f73d None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:01.868077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:01.876124 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c8e373-c353-49e5-9c06-f43fe2b43252 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:01.876445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c8e373-c353-49e5-9c06-f43fe2b43252 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:01.876722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c8e373-c353-49e5-9c06-f43fe2b43252 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:01.878197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 552/1087] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:03:01.883053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a32791f-f116-4019-af8a-efeef5f306e6 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:01.883336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a32791f-f116-4019-af8a-efeef5f306e6 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:01.883607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a32791f-f116-4019-af8a-efeef5f306e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:01.889510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24ef9b0-bf08-444e-8470-d96926eca124 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:01.890132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24ef9b0-bf08-444e-8470-d96926eca124 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:01.890413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24ef9b0-bf08-444e-8470-d96926eca124 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:01.890685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24ef9b0-bf08-444e-8470-d96926eca124 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:01.901364 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-9a32791f-f116-4019-af8a-efeef5f306e6 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:01.901590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24ef9b0-bf08-444e-8470-d96926eca124 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:03:01.902533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-9a32791f-f116-4019-af8a-efeef5f306e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-d24ef9b0-bf08-444e-8470-d96926eca124 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:03:01.906272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:03:01.907442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:03:01.908367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:03:01.908367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:03:01.908367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:03:01.908367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:03:01.908367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:03:01.908367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:03:01.908367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:03:01.908367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:03:01.908367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:03:01.908367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:03:01.908367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 28 12:03:01.908367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:03:01.908367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 536/1088] 10.222.0.26 () {56 vars in 1120 bytes} [Wed Sep 28 12:03:01 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:03:01.913851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24ef9b0-bf08-444e-8470-d96926eca124 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:01.979880 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 553/1089] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:03:02.014022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24ef9b0-bf08-444e-8470-d96926eca124 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:02.046100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48545245-9a9b-4670-a4e9-d8f423236aa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:02.046690 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48545245-9a9b-4670-a4e9-d8f423236aa5 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:02.046967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48545245-9a9b-4670-a4e9-d8f423236aa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:02.047238 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48545245-9a9b-4670-a4e9-d8f423236aa5 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:02.047893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48545245-9a9b-4670-a4e9-d8f423236aa5 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:02.048193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48545245-9a9b-4670-a4e9-d8f423236aa5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:02.050233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48545245-9a9b-4670-a4e9-d8f423236aa5 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:02.057767 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 537/1090] 10.222.0.26 () {56 vars in 1115 bytes} [Wed Sep 28 12:03:01 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:03:02.063602 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a32791f-f116-4019-af8a-efeef5f306e6 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:02.087221 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:02.160881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a32791f-f116-4019-af8a-efeef5f306e6 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:02.198956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfa38a7-7eaa-4028-856f-62975d5abc55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:02.199577 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfa38a7-7eaa-4028-856f-62975d5abc55 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:02.199856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfa38a7-7eaa-4028-856f-62975d5abc55 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:02.200128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfa38a7-7eaa-4028-856f-62975d5abc55 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:02.209826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbfa38a7-7eaa-4028-856f-62975d5abc55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:02.211259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbfa38a7-7eaa-4028-856f-62975d5abc55 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-dbfa38a7-7eaa-4028-856f-62975d5abc55 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:03:02.226605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:03:02.227701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:03:02.228832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:03:02.228832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:03:02.228832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:03:02.232485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 554/1091] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:03:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:03:02.233687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48545245-9a9b-4670-a4e9-d8f423236aa5 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:02.249691 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfa38a7-7eaa-4028-856f-62975d5abc55 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:02.268571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:02.324537 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:02.334266 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48545245-9a9b-4670-a4e9-d8f423236aa5 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:02.347878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfa38a7-7eaa-4028-856f-62975d5abc55 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:02.368220 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6330340-3e80-49c4-a641-279c53256c36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:02.368848 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6330340-3e80-49c4-a641-279c53256c36 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:02.369121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6330340-3e80-49c4-a641-279c53256c36 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:02.369390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6330340-3e80-49c4-a641-279c53256c36 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:02.370024 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6330340-3e80-49c4-a641-279c53256c36 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:02.370319 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6330340-3e80-49c4-a641-279c53256c36 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:02.371675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6330340-3e80-49c4-a641-279c53256c36 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:02.380585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5f7c1c-0d39-4f11-8360-05b7cf8ce00c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:02.381203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5f7c1c-0d39-4f11-8360-05b7cf8ce00c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:02.381478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5f7c1c-0d39-4f11-8360-05b7cf8ce00c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:02.381750 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5f7c1c-0d39-4f11-8360-05b7cf8ce00c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:02.383171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 538/1092] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:03:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:02.389052 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6330340-3e80-49c4-a641-279c53256c36 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:02.391978 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5f7c1c-0d39-4f11-8360-05b7cf8ce00c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-1c5f7c1c-0d39-4f11-8360-05b7cf8ce00c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:03:02.397254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:03:02.398186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:03:02.399226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:03:02.399226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:03:02.399226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:03:02.399226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:03:02.399226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:03:02.399226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:03:02.399226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:03:02.399226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:03:02.399226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:03:02.399226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:03:02.399226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:03:02.399226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:03:02.399226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 555/1093] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:03:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:03:02.404391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5f7c1c-0d39-4f11-8360-05b7cf8ce00c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:02.479903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:02.489866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6330340-3e80-49c4-a641-279c53256c36 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:02.504493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5f7c1c-0d39-4f11-8360-05b7cf8ce00c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:02.524031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8f9976-ba76-4294-b07e-8a1aefd6bd57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:02.524623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8f9976-ba76-4294-b07e-8a1aefd6bd57 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:02.524903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8f9976-ba76-4294-b07e-8a1aefd6bd57 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:02.525170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8f9976-ba76-4294-b07e-8a1aefd6bd57 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:02.531954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8f9976-ba76-4294-b07e-8a1aefd6bd57 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:02.534003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8f9976-ba76-4294-b07e-8a1aefd6bd57 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:02.538861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c44e960-26e9-4e86-b5aa-057833f9b1dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:02.539449 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c44e960-26e9-4e86-b5aa-057833f9b1dd None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:02.539725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c44e960-26e9-4e86-b5aa-057833f9b1dd None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:02.540002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c44e960-26e9-4e86-b5aa-057833f9b1dd None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:02.540688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c44e960-26e9-4e86-b5aa-057833f9b1dd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:02.540988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c44e960-26e9-4e86-b5aa-057833f9b1dd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:02.542395 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c44e960-26e9-4e86-b5aa-057833f9b1dd None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:02.544481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-7c8f9976-ba76-4294-b07e-8a1aefd6bd57 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:03:02.545234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:03:02.546971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:03:02.548327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:03:02.548327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:03:02.548327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:03:02.548327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:03:02.548327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:03:02.548327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:03:02.548327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:03:02.548327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:03:02.548327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 28 12:03:02.548327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:03:02.548327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 539/1094] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Sep 28 12:03:02 2022] GET /identity/v3/users/nova => generated 82 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:03:02.551834 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 556/1095] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:03:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:03:02.552148 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8f9976-ba76-4294-b07e-8a1aefd6bd57 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:02.557815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c44e960-26e9-4e86-b5aa-057833f9b1dd None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:02.651583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8f9976-ba76-4294-b07e-8a1aefd6bd57 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:02.654425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c44e960-26e9-4e86-b5aa-057833f9b1dd None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:02.684714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada1aa14-33ae-4a3e-848c-559b106597e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:02.685309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada1aa14-33ae-4a3e-848c-559b106597e2 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:02.685586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada1aa14-33ae-4a3e-848c-559b106597e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:02.685856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada1aa14-33ae-4a3e-848c-559b106597e2 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:02.686570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada1aa14-33ae-4a3e-848c-559b106597e2 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:02.686870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada1aa14-33ae-4a3e-848c-559b106597e2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:02.688088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f02905-2d19-4d63-9655-b523e4ebf49d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:02.688284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada1aa14-33ae-4a3e-848c-559b106597e2 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:02.688721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f02905-2d19-4d63-9655-b523e4ebf49d None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:02.689001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f02905-2d19-4d63-9655-b523e4ebf49d None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:02.689270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f02905-2d19-4d63-9655-b523e4ebf49d None admin] PATH_INFO: `/v3/users/89df2cea6cb14b4db32b364990dacbf1` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:02.690411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f02905-2d19-4d63-9655-b523e4ebf49d None admin] RBAC: Authorizing `identity:get_user(user_id=89df2cea6cb14b4db32b364990dacbf1)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:02.691789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f02905-2d19-4d63-9655-b523e4ebf49d None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:02.693364 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 557/1096] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:03:02 2022] GET /identity/v3/users/89df2cea6cb14b4db32b364990dacbf1 => generated 285 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:02.699002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f02905-2d19-4d63-9655-b523e4ebf49d None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:02.711004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 540/1097] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Sep 28 12:03:02 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:03:02.711971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada1aa14-33ae-4a3e-848c-559b106597e2 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:02.726854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:02.761904 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:02.797123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f02905-2d19-4d63-9655-b523e4ebf49d None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:02.809860 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada1aa14-33ae-4a3e-848c-559b106597e2 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:02.829921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e41b03-7f4b-4659-9af8-56dfe34d3cba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:02.831059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e41b03-7f4b-4659-9af8-56dfe34d3cba None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:02.831336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e41b03-7f4b-4659-9af8-56dfe34d3cba None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:02.831608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e41b03-7f4b-4659-9af8-56dfe34d3cba None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:02.843355 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b114a31d-16b6-4b17-a352-7ba4355817a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:02.843928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b114a31d-16b6-4b17-a352-7ba4355817a6 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:02.844203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b114a31d-16b6-4b17-a352-7ba4355817a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:02.844500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b114a31d-16b6-4b17-a352-7ba4355817a6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:02.897914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e41b03-7f4b-4659-9af8-56dfe34d3cba None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:02.899287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e41b03-7f4b-4659-9af8-56dfe34d3cba None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:02.909433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b114a31d-16b6-4b17-a352-7ba4355817a6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:02.910869 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b114a31d-16b6-4b17-a352-7ba4355817a6 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:02.923834 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:02.962510 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:02.998908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 558/1098] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:03:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:03:03.004823 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e41b03-7f4b-4659-9af8-56dfe34d3cba None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:03.010330 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 541/1099] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:03:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:03:03.015679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b114a31d-16b6-4b17-a352-7ba4355817a6 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:03.104234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e41b03-7f4b-4659-9af8-56dfe34d3cba None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:03.110296 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b114a31d-16b6-4b17-a352-7ba4355817a6 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:03.124387 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:03.138662 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30e12a3-b58d-43a7-9f4e-b72c9ebe9d7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:03.139288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30e12a3-b58d-43a7-9f4e-b72c9ebe9d7c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:03.139568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30e12a3-b58d-43a7-9f4e-b72c9ebe9d7c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:03.139842 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30e12a3-b58d-43a7-9f4e-b72c9ebe9d7c None admin] PATH_INFO: `/v3/projects/28a0c86e6c374712b465b9b45443db89` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:03.140988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30e12a3-b58d-43a7-9f4e-b72c9ebe9d7c None admin] RBAC: Authorizing `identity:get_project(project_id=28a0c86e6c374712b465b9b45443db89)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:03.142411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30e12a3-b58d-43a7-9f4e-b72c9ebe9d7c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:03.144572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 559/1100] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:03:03 2022] GET /identity/v3/projects/28a0c86e6c374712b465b9b45443db89 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:03:03.146465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9782f385-3ddf-4524-9e07-110a6548b596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:03.147083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9782f385-3ddf-4524-9e07-110a6548b596 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:03.147358 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9782f385-3ddf-4524-9e07-110a6548b596 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:03.147626 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9782f385-3ddf-4524-9e07-110a6548b596 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:03.150706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30e12a3-b58d-43a7-9f4e-b72c9ebe9d7c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:03.158368 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9782f385-3ddf-4524-9e07-110a6548b596 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:03.160407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9782f385-3ddf-4524-9e07-110a6548b596 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-9782f385-3ddf-4524-9e07-110a6548b596 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:03:03.174836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:03:03.175764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:03:03.176721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:03:03.176721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:03:03.176721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:03:03.176721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 542/1101] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:03:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:03:03.181417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9782f385-3ddf-4524-9e07-110a6548b596 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:03.186048 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:03.252061 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30e12a3-b58d-43a7-9f4e-b72c9ebe9d7c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:03.277451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9782f385-3ddf-4524-9e07-110a6548b596 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:03.285377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f060e76-5add-45d4-8c25-91bbcccf597d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:03.285969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f060e76-5add-45d4-8c25-91bbcccf597d None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:03.286240 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f060e76-5add-45d4-8c25-91bbcccf597d None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:03.286520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f060e76-5add-45d4-8c25-91bbcccf597d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:03.287238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f060e76-5add-45d4-8c25-91bbcccf597d None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=28a0c86e6c374712b465b9b45443db89, user.id=89df2cea6cb14b4db32b364990dacbf1) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:03.287542 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f060e76-5add-45d4-8c25-91bbcccf597d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:03.288974 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f060e76-5add-45d4-8c25-91bbcccf597d None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:03.294531 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 560/1102] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:03:03 2022] GET /identity/v3/role_assignments?user.id=89df2cea6cb14b4db32b364990dacbf1&scope.project.id=28a0c86e6c374712b465b9b45443db89&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:03:03.310633 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba017b0a-2964-4619-9cac-d179fd8fea6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:03.311250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba017b0a-2964-4619-9cac-d179fd8fea6f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:03.311525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba017b0a-2964-4619-9cac-d179fd8fea6f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:03.311791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba017b0a-2964-4619-9cac-d179fd8fea6f None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:03.312484 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba017b0a-2964-4619-9cac-d179fd8fea6f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:03.312787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba017b0a-2964-4619-9cac-d179fd8fea6f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:03.314198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba017b0a-2964-4619-9cac-d179fd8fea6f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:03.327389 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 543/1103] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:03:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:03:03.333970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f060e76-5add-45d4-8c25-91bbcccf597d None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:03.435010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:03.439485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f060e76-5add-45d4-8c25-91bbcccf597d None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:03.474800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42e5157-04b9-43d9-9f3e-7263402648c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:03.475423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42e5157-04b9-43d9-9f3e-7263402648c6 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:03.475700 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42e5157-04b9-43d9-9f3e-7263402648c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:03.475971 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42e5157-04b9-43d9-9f3e-7263402648c6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:03.485450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42e5157-04b9-43d9-9f3e-7263402648c6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:03.487424 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42e5157-04b9-43d9-9f3e-7263402648c6 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-b42e5157-04b9-43d9-9f3e-7263402648c6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:03:03.499274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:03:03.500161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:03:03.501175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:03:03.501175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:03:03.501175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:03:03.501175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:03:03.501175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:03:03.501175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:03:03.501175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:03:03.501175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:03:03.501175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 561/1104] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:03:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:03:03.506961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba017b0a-2964-4619-9cac-d179fd8fea6f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:03.532179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:03.606503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba017b0a-2964-4619-9cac-d179fd8fea6f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:03.639627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6184a18e-aebc-4b01-982a-cd81c931d24b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:03.640225 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6184a18e-aebc-4b01-982a-cd81c931d24b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:03.640533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6184a18e-aebc-4b01-982a-cd81c931d24b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:03.640804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6184a18e-aebc-4b01-982a-cd81c931d24b None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:03.641493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6184a18e-aebc-4b01-982a-cd81c931d24b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:03.641796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6184a18e-aebc-4b01-982a-cd81c931d24b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:03.643218 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6184a18e-aebc-4b01-982a-cd81c931d24b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:03.654338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 544/1105] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:03:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:03.660990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42e5157-04b9-43d9-9f3e-7263402648c6 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:03.756645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42e5157-04b9-43d9-9f3e-7263402648c6 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:03.788530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8abd499-39e0-4cd9-bcdf-d8420d1a3005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:03.789121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8abd499-39e0-4cd9-bcdf-d8420d1a3005 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:03.789394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8abd499-39e0-4cd9-bcdf-d8420d1a3005 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:03.789661 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8abd499-39e0-4cd9-bcdf-d8420d1a3005 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:03.790375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8abd499-39e0-4cd9-bcdf-d8420d1a3005 None admin] RBAC: Adding query filter params (role.id=d1813115b0cf4d2d905502119199b5db, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=8bf969348e7a4158acf8fb789430b0e6) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:03.790676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8abd499-39e0-4cd9-bcdf-d8420d1a3005 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:03.792088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8abd499-39e0-4cd9-bcdf-d8420d1a3005 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:03.797704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 562/1106] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:03:03 2022] GET /identity/v3/role_assignments?user.id=8bf969348e7a4158acf8fb789430b0e6&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=d1813115b0cf4d2d905502119199b5db => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:03:05.473572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfc107a-f819-4413-acb0-b3d7cf2db388 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:05.474126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfc107a-f819-4413-acb0-b3d7cf2db388 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:05.474126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfc107a-f819-4413-acb0-b3d7cf2db388 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:05.475457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 545/1107] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:05.480666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804c8ac8-742f-4b34-8ed9-5631662cf871 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:05.481091 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804c8ac8-742f-4b34-8ed9-5631662cf871 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:05.481225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804c8ac8-742f-4b34-8ed9-5631662cf871 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:05.498867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-804c8ac8-742f-4b34-8ed9-5631662cf871 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:05.500074 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-804c8ac8-742f-4b34-8ed9-5631662cf871 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:05.516054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:05.566377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 563/1108] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:03:05.838122 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfc107a-f819-4413-acb0-b3d7cf2db388 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:05.928592 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:05.945672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfc107a-f819-4413-acb0-b3d7cf2db388 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:05.977763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9ba373-2fcc-49cf-ba3f-78256989aafe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:05.978376 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9ba373-2fcc-49cf-ba3f-78256989aafe None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:05.978656 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9ba373-2fcc-49cf-ba3f-78256989aafe None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:05.978929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9ba373-2fcc-49cf-ba3f-78256989aafe None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:05.980031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9ba373-2fcc-49cf-ba3f-78256989aafe None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:05.981495 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9ba373-2fcc-49cf-ba3f-78256989aafe None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:05.983045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 546/1109] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:03:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:05.989033 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804c8ac8-742f-4b34-8ed9-5631662cf871 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:06.083856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804c8ac8-742f-4b34-8ed9-5631662cf871 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:06.115989 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98b89c3-085f-4610-9377-5ce839c0ab36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:06.116721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98b89c3-085f-4610-9377-5ce839c0ab36 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:06.117093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98b89c3-085f-4610-9377-5ce839c0ab36 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:06.117459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98b89c3-085f-4610-9377-5ce839c0ab36 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:06.118107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98b89c3-085f-4610-9377-5ce839c0ab36 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:06.119952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98b89c3-085f-4610-9377-5ce839c0ab36 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:06.144882 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 564/1110] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:03:05 2022] POST /identity/v3/users => generated 286 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:03:08.309484 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52a52fd-44e2-4eb5-b83e-9be48f33f48b None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:08.310031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52a52fd-44e2-4eb5-b83e-9be48f33f48b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:08.310031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52a52fd-44e2-4eb5-b83e-9be48f33f48b None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:08.311369 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 547/1111] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:08.316560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdd59d9-dff8-4c3f-b0e2-5ca692cc7054 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:08.316978 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdd59d9-dff8-4c3f-b0e2-5ca692cc7054 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:08.317112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdd59d9-dff8-4c3f-b0e2-5ca692cc7054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:08.334703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-8bdd59d9-dff8-4c3f-b0e2-5ca692cc7054 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:08.335893 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8bdd59d9-dff8-4c3f-b0e2-5ca692cc7054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:08.338068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:08.401620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 565/1112] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:08.673259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52a52fd-44e2-4eb5-b83e-9be48f33f48b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:08.772575 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52a52fd-44e2-4eb5-b83e-9be48f33f48b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:08.810618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d28359-b685-4adb-8908-12813c0f2b30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:08.811251 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d28359-b685-4adb-8908-12813c0f2b30 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:08.811535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d28359-b685-4adb-8908-12813c0f2b30 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:08.811813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d28359-b685-4adb-8908-12813c0f2b30 None admin] PATH_INFO: `/v3/roles/a3902a185ac144fb8e24f4e9feb039ed` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:08.813254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d28359-b685-4adb-8908-12813c0f2b30 None admin] RBAC: Authorizing `identity:get_role(role_id=a3902a185ac144fb8e24f4e9feb039ed)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:08.814938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d28359-b685-4adb-8908-12813c0f2b30 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:08.817634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 548/1113] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:03:08 2022] GET /identity/v3/roles/a3902a185ac144fb8e24f4e9feb039ed => generated 223 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:08.823358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdd59d9-dff8-4c3f-b0e2-5ca692cc7054 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:08.918146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdd59d9-dff8-4c3f-b0e2-5ca692cc7054 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:08.950299 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09db8af9-4635-485c-b36c-9f26cab28ea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:08.950858 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09db8af9-4635-485c-b36c-9f26cab28ea2 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:08.951135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09db8af9-4635-485c-b36c-9f26cab28ea2 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:08.951405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09db8af9-4635-485c-b36c-9f26cab28ea2 None admin] PATH_INFO: `/v3/users/5312fe8d83cd45ec94882414f3fbdeee` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:08.958755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:08.964790 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09db8af9-4635-485c-b36c-9f26cab28ea2 None admin] RBAC: Authorizing `identity:get_user(user_id=5312fe8d83cd45ec94882414f3fbdeee)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:08.966199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09db8af9-4635-485c-b36c-9f26cab28ea2 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:08.967761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 566/1114] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:03:08 2022] GET /identity/v3/users/5312fe8d83cd45ec94882414f3fbdeee => generated 286 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:08.974113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d28359-b685-4adb-8908-12813c0f2b30 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:08.995478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:09.070078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d28359-b685-4adb-8908-12813c0f2b30 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:09.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e243724-29b3-4cfd-b5d2-3556156b1b3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:09.102793 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e243724-29b3-4cfd-b5d2-3556156b1b3c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:09.103073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e243724-29b3-4cfd-b5d2-3556156b1b3c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:09.103347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e243724-29b3-4cfd-b5d2-3556156b1b3c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:09.178911 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e243724-29b3-4cfd-b5d2-3556156b1b3c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:09.180386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e243724-29b3-4cfd-b5d2-3556156b1b3c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:09.199469 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:09.277962 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 549/1115] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:03:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:03:09.284541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09db8af9-4635-485c-b36c-9f26cab28ea2 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:09.378437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09db8af9-4635-485c-b36c-9f26cab28ea2 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:09.411142 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896bcaf0-27b7-41af-a20c-d8dbd516a15e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:09.411753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896bcaf0-27b7-41af-a20c-d8dbd516a15e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:09.412031 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896bcaf0-27b7-41af-a20c-d8dbd516a15e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:09.412337 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896bcaf0-27b7-41af-a20c-d8dbd516a15e None admin] PATH_INFO: `/v3/projects/28a0c86e6c374712b465b9b45443db89` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:09.413456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896bcaf0-27b7-41af-a20c-d8dbd516a15e None admin] RBAC: Authorizing `identity:get_project(project_id=28a0c86e6c374712b465b9b45443db89)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:09.414869 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896bcaf0-27b7-41af-a20c-d8dbd516a15e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:09.416535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 567/1116] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:03:09 2022] GET /identity/v3/projects/28a0c86e6c374712b465b9b45443db89 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:09.422788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e243724-29b3-4cfd-b5d2-3556156b1b3c None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:09.521885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e243724-29b3-4cfd-b5d2-3556156b1b3c None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:09.540662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:09.554103 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fdc82e-0dc4-4cbe-8b82-fbd513fec159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:09.554688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fdc82e-0dc4-4cbe-8b82-fbd513fec159 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:09.554961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fdc82e-0dc4-4cbe-8b82-fbd513fec159 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:09.555226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fdc82e-0dc4-4cbe-8b82-fbd513fec159 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:09.555937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fdc82e-0dc4-4cbe-8b82-fbd513fec159 None admin] RBAC: Adding query filter params (role.id=a3902a185ac144fb8e24f4e9feb039ed, scope.project.id=28a0c86e6c374712b465b9b45443db89, user.id=5312fe8d83cd45ec94882414f3fbdeee) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:09.556237 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fdc82e-0dc4-4cbe-8b82-fbd513fec159 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:09.557676 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fdc82e-0dc4-4cbe-8b82-fbd513fec159 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:09.562758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 550/1117] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:03:09 2022] GET /identity/v3/role_assignments?user.id=5312fe8d83cd45ec94882414f3fbdeee&scope.project.id=28a0c86e6c374712b465b9b45443db89&role.id=a3902a185ac144fb8e24f4e9feb039ed => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:11.675959 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa76c653-c922-40a6-a609-69d8714b79df None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:11.676511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa76c653-c922-40a6-a609-69d8714b79df None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:11.676620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa76c653-c922-40a6-a609-69d8714b79df None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:11.677867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 568/1118] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:11.683069 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e64fd4-07c3-4357-a0e5-796f76d9d6df None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:11.683506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e64fd4-07c3-4357-a0e5-796f76d9d6df None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:11.683597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e64fd4-07c3-4357-a0e5-796f76d9d6df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:11.701628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-05e64fd4-07c3-4357-a0e5-796f76d9d6df None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:11.702796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-05e64fd4-07c3-4357-a0e5-796f76d9d6df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:11.765202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 551/1119] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:12.032282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa76c653-c922-40a6-a609-69d8714b79df None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:12.096583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:12.133390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa76c653-c922-40a6-a609-69d8714b79df None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:12.166116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3540f42-aabe-457c-b371-3493f9e736ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:12.166727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3540f42-aabe-457c-b371-3493f9e736ec None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:12.167004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3540f42-aabe-457c-b371-3493f9e736ec None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:12.167274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3540f42-aabe-457c-b371-3493f9e736ec None admin] PATH_INFO: `/v3/roles/a3902a185ac144fb8e24f4e9feb039ed` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:12.168677 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3540f42-aabe-457c-b371-3493f9e736ec None admin] RBAC: Authorizing `identity:get_role(role_id=a3902a185ac144fb8e24f4e9feb039ed)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:12.170020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3540f42-aabe-457c-b371-3493f9e736ec None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:12.171318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 569/1120] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:03:12 2022] GET /identity/v3/roles/a3902a185ac144fb8e24f4e9feb039ed => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:12.177114 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e64fd4-07c3-4357-a0e5-796f76d9d6df None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:12.270309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e64fd4-07c3-4357-a0e5-796f76d9d6df None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:12.283952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:12.303131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771114bc-3cd8-4cf1-b81d-058e81afb009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:12.303736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771114bc-3cd8-4cf1-b81d-058e81afb009 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:12.304010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771114bc-3cd8-4cf1-b81d-058e81afb009 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:12.304276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771114bc-3cd8-4cf1-b81d-058e81afb009 None admin] PATH_INFO: `/v3/users/5312fe8d83cd45ec94882414f3fbdeee` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:12.305405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771114bc-3cd8-4cf1-b81d-058e81afb009 None admin] RBAC: Authorizing `identity:get_user(user_id=5312fe8d83cd45ec94882414f3fbdeee)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:12.306824 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771114bc-3cd8-4cf1-b81d-058e81afb009 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:12.308367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 552/1121] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:03:12 2022] GET /identity/v3/users/5312fe8d83cd45ec94882414f3fbdeee => generated 286 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:12.313937 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3540f42-aabe-457c-b371-3493f9e736ec None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:12.410010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3540f42-aabe-457c-b371-3493f9e736ec None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:12.436749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:12.441791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a47d298-4a84-4d39-9640-8a8d43a9ce60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:12.442397 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a47d298-4a84-4d39-9640-8a8d43a9ce60 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:12.442673 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a47d298-4a84-4d39-9640-8a8d43a9ce60 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:12.442941 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a47d298-4a84-4d39-9640-8a8d43a9ce60 None admin] PATH_INFO: `/v3/projects/28a0c86e6c374712b465b9b45443db89` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:12.444038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a47d298-4a84-4d39-9640-8a8d43a9ce60 None admin] RBAC: Authorizing `identity:get_project(project_id=28a0c86e6c374712b465b9b45443db89)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:12.445481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a47d298-4a84-4d39-9640-8a8d43a9ce60 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:12.447068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 570/1122] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:03:12 2022] GET /identity/v3/projects/28a0c86e6c374712b465b9b45443db89 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:12.452951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771114bc-3cd8-4cf1-b81d-058e81afb009 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:12.548945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771114bc-3cd8-4cf1-b81d-058e81afb009 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:12.581738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635459ea-8779-428d-8cbc-52c840db097f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:12.582371 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635459ea-8779-428d-8cbc-52c840db097f None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:12.582643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635459ea-8779-428d-8cbc-52c840db097f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:12.582909 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635459ea-8779-428d-8cbc-52c840db097f None admin] PATH_INFO: `/v3/projects/28a0c86e6c374712b465b9b45443db89/users/5312fe8d83cd45ec94882414f3fbdeee/roles/a3902a185ac144fb8e24f4e9feb039ed` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:12.584989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635459ea-8779-428d-8cbc-52c840db097f None admin] RBAC: Authorizing `identity:create_grant(project_id=28a0c86e6c374712b465b9b45443db89, user_id=5312fe8d83cd45ec94882414f3fbdeee, role_id=a3902a185ac144fb8e24f4e9feb039ed)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:12.586405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635459ea-8779-428d-8cbc-52c840db097f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:12.600087 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 553/1123] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:03:12 2022] PUT /identity/v3/projects/28a0c86e6c374712b465b9b45443db89/users/5312fe8d83cd45ec94882414f3fbdeee/roles/a3902a185ac144fb8e24f4e9feb039ed => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:03:14.717356 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d76e920-2e5f-411e-b700-569ed96d2671 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:14.717898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d76e920-2e5f-411e-b700-569ed96d2671 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:14.717898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d76e920-2e5f-411e-b700-569ed96d2671 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:14.719218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 571/1124] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:14.725013 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739745a7-978f-4aab-a163-b7432cb4deac None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:14.726276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739745a7-978f-4aab-a163-b7432cb4deac None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:14.726589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739745a7-978f-4aab-a163-b7432cb4deac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:14.744353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-739745a7-978f-4aab-a163-b7432cb4deac None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:14.745487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-739745a7-978f-4aab-a163-b7432cb4deac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:14.779426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:14.810417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 554/1125] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:15.077686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d76e920-2e5f-411e-b700-569ed96d2671 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:15.174575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d76e920-2e5f-411e-b700-569ed96d2671 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:15.206182 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e397ae77-ff84-40d3-b8ce-a5525b4b013e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:15.206780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e397ae77-ff84-40d3-b8ce-a5525b4b013e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:15.207053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e397ae77-ff84-40d3-b8ce-a5525b4b013e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:15.207319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e397ae77-ff84-40d3-b8ce-a5525b4b013e None admin] PATH_INFO: `/v3/roles/a3902a185ac144fb8e24f4e9feb039ed` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:15.208704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e397ae77-ff84-40d3-b8ce-a5525b4b013e None admin] RBAC: Authorizing `identity:get_role(role_id=a3902a185ac144fb8e24f4e9feb039ed)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:15.210108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e397ae77-ff84-40d3-b8ce-a5525b4b013e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:15.211381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 572/1126] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:03:15 2022] GET /identity/v3/roles/a3902a185ac144fb8e24f4e9feb039ed => generated 223 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:15.217194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739745a7-978f-4aab-a163-b7432cb4deac None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:15.311279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739745a7-978f-4aab-a163-b7432cb4deac None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:15.344548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fc9bd8-40f0-4ec4-b0a0-61b07f41306b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:15.345106 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fc9bd8-40f0-4ec4-b0a0-61b07f41306b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:15.345386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fc9bd8-40f0-4ec4-b0a0-61b07f41306b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:15.345737 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fc9bd8-40f0-4ec4-b0a0-61b07f41306b None admin] PATH_INFO: `/v3/users/5312fe8d83cd45ec94882414f3fbdeee` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:15.346848 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fc9bd8-40f0-4ec4-b0a0-61b07f41306b None admin] RBAC: Authorizing `identity:get_user(user_id=5312fe8d83cd45ec94882414f3fbdeee)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:15.348272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fc9bd8-40f0-4ec4-b0a0-61b07f41306b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:15.349830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 555/1127] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:03:15 2022] GET /identity/v3/users/5312fe8d83cd45ec94882414f3fbdeee => generated 286 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:15.355310 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e397ae77-ff84-40d3-b8ce-a5525b4b013e None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:15.418323 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:15.450490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e397ae77-ff84-40d3-b8ce-a5525b4b013e None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:15.481647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d45eeca-8890-493e-92d9-58bfa24fdedd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:15.482236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d45eeca-8890-493e-92d9-58bfa24fdedd None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:15.482511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d45eeca-8890-493e-92d9-58bfa24fdedd None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:15.482778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d45eeca-8890-493e-92d9-58bfa24fdedd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:15.547467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d45eeca-8890-493e-92d9-58bfa24fdedd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:15.549530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d45eeca-8890-493e-92d9-58bfa24fdedd None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:15.609278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:15.641438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 573/1128] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:03:15 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 287 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:03:15.647883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fc9bd8-40f0-4ec4-b0a0-61b07f41306b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:15.687297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:15.743696 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fc9bd8-40f0-4ec4-b0a0-61b07f41306b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:15.776486 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fa02a1-4605-4794-a097-41d8fec297ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:15.777103 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fa02a1-4605-4794-a097-41d8fec297ad None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:15.777380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fa02a1-4605-4794-a097-41d8fec297ad None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:15.777651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fa02a1-4605-4794-a097-41d8fec297ad None admin] PATH_INFO: `/v3/projects/28a0c86e6c374712b465b9b45443db89` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:15.778761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fa02a1-4605-4794-a097-41d8fec297ad None admin] RBAC: Authorizing `identity:get_project(project_id=28a0c86e6c374712b465b9b45443db89)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:15.780187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fa02a1-4605-4794-a097-41d8fec297ad None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:15.781827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 556/1129] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:03:15 2022] GET /identity/v3/projects/28a0c86e6c374712b465b9b45443db89 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:15.788032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d45eeca-8890-493e-92d9-58bfa24fdedd None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:15.885028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d45eeca-8890-493e-92d9-58bfa24fdedd None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:15.917042 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf34a48-2654-4b47-9141-ee549db163d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:15.917630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf34a48-2654-4b47-9141-ee549db163d2 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:15.917906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf34a48-2654-4b47-9141-ee549db163d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:15.918175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf34a48-2654-4b47-9141-ee549db163d2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:15.918891 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf34a48-2654-4b47-9141-ee549db163d2 None admin] RBAC: Adding query filter params (role.id=a3902a185ac144fb8e24f4e9feb039ed, scope.project.id=28a0c86e6c374712b465b9b45443db89, user.id=5312fe8d83cd45ec94882414f3fbdeee) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:15.919196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf34a48-2654-4b47-9141-ee549db163d2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:15.920632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf34a48-2654-4b47-9141-ee549db163d2 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:15.926686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 574/1130] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:03:15 2022] GET /identity/v3/role_assignments?user.id=5312fe8d83cd45ec94882414f3fbdeee&scope.project.id=28a0c86e6c374712b465b9b45443db89&role.id=a3902a185ac144fb8e24f4e9feb039ed => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:18.068721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7904c01-26e6-47a6-86ac-98a0de0e02da None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:18.069278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7904c01-26e6-47a6-86ac-98a0de0e02da None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:18.069278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7904c01-26e6-47a6-86ac-98a0de0e02da None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:18.070607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 557/1131] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:18.076622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e04b24-5661-4838-b7c5-fbc0e5abb8a9 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:18.077047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e04b24-5661-4838-b7c5-fbc0e5abb8a9 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:18.077173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e04b24-5661-4838-b7c5-fbc0e5abb8a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:18.096248 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-95e04b24-5661-4838-b7c5-fbc0e5abb8a9 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:18.097448 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-95e04b24-5661-4838-b7c5-fbc0e5abb8a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:18.121899 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:18.161308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 575/1132] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:18.431771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7904c01-26e6-47a6-86ac-98a0de0e02da None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:18.530812 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7904c01-26e6-47a6-86ac-98a0de0e02da None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:18.539695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:18.564730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8b16f4-1e90-4e0f-b54f-4544ace7778a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:18.565341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8b16f4-1e90-4e0f-b54f-4544ace7778a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:18.565619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8b16f4-1e90-4e0f-b54f-4544ace7778a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:18.565889 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8b16f4-1e90-4e0f-b54f-4544ace7778a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:18.567002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8b16f4-1e90-4e0f-b54f-4544ace7778a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:18.568465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8b16f4-1e90-4e0f-b54f-4544ace7778a None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:18.570012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 558/1133] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:03:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:18.575861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e04b24-5661-4838-b7c5-fbc0e5abb8a9 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:18.670279 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e04b24-5661-4838-b7c5-fbc0e5abb8a9 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:18.702264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6960bf-3c1e-4e67-a283-492ed140e717 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:18.702800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6960bf-3c1e-4e67-a283-492ed140e717 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:18.703077 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6960bf-3c1e-4e67-a283-492ed140e717 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:18.703348 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6960bf-3c1e-4e67-a283-492ed140e717 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:18.703999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6960bf-3c1e-4e67-a283-492ed140e717 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:18.706025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6960bf-3c1e-4e67-a283-492ed140e717 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:18.731326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 576/1134] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Sep 28 12:03:18 2022] POST /identity/v3/projects => generated 311 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:03:20.883209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d344e848-8eba-40ef-b597-ff9bff830d93 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:20.883821 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d344e848-8eba-40ef-b597-ff9bff830d93 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:20.883821 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d344e848-8eba-40ef-b597-ff9bff830d93 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:20.885095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 559/1135] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:20.890332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115c07f7-dda2-4fa8-a11a-7d4168989e05 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:20.890732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115c07f7-dda2-4fa8-a11a-7d4168989e05 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:20.890871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115c07f7-dda2-4fa8-a11a-7d4168989e05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:20.909051 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-115c07f7-dda2-4fa8-a11a-7d4168989e05 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:20.910214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-115c07f7-dda2-4fa8-a11a-7d4168989e05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:20.918924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:20.974931 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 577/1136] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:21.243303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d344e848-8eba-40ef-b597-ff9bff830d93 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:21.340424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d344e848-8eba-40ef-b597-ff9bff830d93 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:21.372898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d331822-c9a5-43e3-9ad6-b16a825cbaa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:21.373503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d331822-c9a5-43e3-9ad6-b16a825cbaa0 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:21.373776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d331822-c9a5-43e3-9ad6-b16a825cbaa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:21.374042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d331822-c9a5-43e3-9ad6-b16a825cbaa0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:21.375137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d331822-c9a5-43e3-9ad6-b16a825cbaa0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:21.376583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d331822-c9a5-43e3-9ad6-b16a825cbaa0 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:21.378117 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 560/1137] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:03:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:21.383822 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115c07f7-dda2-4fa8-a11a-7d4168989e05 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:21.478061 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115c07f7-dda2-4fa8-a11a-7d4168989e05 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:21.510693 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62afe5bc-3cc6-492d-90b1-e15af9936b8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:21.511232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62afe5bc-3cc6-492d-90b1-e15af9936b8a None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:21.511508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62afe5bc-3cc6-492d-90b1-e15af9936b8a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:21.511779 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62afe5bc-3cc6-492d-90b1-e15af9936b8a None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:21.512461 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62afe5bc-3cc6-492d-90b1-e15af9936b8a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:21.513871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62afe5bc-3cc6-492d-90b1-e15af9936b8a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:21.530604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:21.536396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 578/1138] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:03:21 2022] POST /identity/v3/users => generated 286 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:03:23.702280 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d746fd-fa8f-4605-b551-9e7e22ead1b1 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:23.702805 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d746fd-fa8f-4605-b551-9e7e22ead1b1 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:23.702805 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d746fd-fa8f-4605-b551-9e7e22ead1b1 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:23.704136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 561/1139] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:23.709504 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a35b1a6-d6d7-4344-b134-77e60556e1ea None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:23.709898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a35b1a6-d6d7-4344-b134-77e60556e1ea None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:23.710053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a35b1a6-d6d7-4344-b134-77e60556e1ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:23.727861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4a35b1a6-d6d7-4344-b134-77e60556e1ea None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:23.729068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4a35b1a6-d6d7-4344-b134-77e60556e1ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:23.791986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 579/1140] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:24.068555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d746fd-fa8f-4605-b551-9e7e22ead1b1 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:24.116579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:24.170760 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d746fd-fa8f-4605-b551-9e7e22ead1b1 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:24.204951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a80612-7bee-4f39-af95-a66f9475a321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:24.205565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a80612-7bee-4f39-af95-a66f9475a321 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:24.205840 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a80612-7bee-4f39-af95-a66f9475a321 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:24.206106 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a80612-7bee-4f39-af95-a66f9475a321 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:24.216166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a80612-7bee-4f39-af95-a66f9475a321 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-70a80612-7bee-4f39-af95-a66f9475a321 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:03:24.220939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:03:24.221906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:03:24.222797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:03:24.222797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:03:24.222797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:03:24.222797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:03:24.222797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:03:24.222797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:03:24.222797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:03:24.222797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:03:24.222797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:03:24.222797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:03:24.222797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:03:24.222797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:03:24.222797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 562/1141] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:03:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:03:24.228213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a35b1a6-d6d7-4344-b134-77e60556e1ea None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:24.323061 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a35b1a6-d6d7-4344-b134-77e60556e1ea None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:24.350169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:24.356083 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc622ad0-dec1-4e27-a83d-1188d94ee911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:24.356702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc622ad0-dec1-4e27-a83d-1188d94ee911 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:24.356983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc622ad0-dec1-4e27-a83d-1188d94ee911 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:24.357260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc622ad0-dec1-4e27-a83d-1188d94ee911 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:24.357918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc622ad0-dec1-4e27-a83d-1188d94ee911 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:24.358222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc622ad0-dec1-4e27-a83d-1188d94ee911 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:24.359556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc622ad0-dec1-4e27-a83d-1188d94ee911 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:24.367844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 580/1142] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:03:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:24.373657 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a80612-7bee-4f39-af95-a66f9475a321 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:24.472527 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a80612-7bee-4f39-af95-a66f9475a321 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:24.474823 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:24.509881 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfebfa29-154e-429a-8be2-842334b8de8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:24.510507 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfebfa29-154e-429a-8be2-842334b8de8b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:24.510789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfebfa29-154e-429a-8be2-842334b8de8b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:24.511062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfebfa29-154e-429a-8be2-842334b8de8b None admin] PATH_INFO: `/v3/users/3748cf2128744fac8c418d98ae6d0d77` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:24.524594 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfebfa29-154e-429a-8be2-842334b8de8b None admin] RBAC: Authorizing `identity:get_user(user_id=3748cf2128744fac8c418d98ae6d0d77)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:24.526042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfebfa29-154e-429a-8be2-842334b8de8b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:24.527618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 563/1143] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:03:24 2022] GET /identity/v3/users/3748cf2128744fac8c418d98ae6d0d77 => generated 286 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:24.533953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc622ad0-dec1-4e27-a83d-1188d94ee911 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:24.628889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc622ad0-dec1-4e27-a83d-1188d94ee911 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:24.660085 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3642901f-6500-481e-9b26-bb47dde60f46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:24.660698 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3642901f-6500-481e-9b26-bb47dde60f46 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:24.660973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3642901f-6500-481e-9b26-bb47dde60f46 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:24.661240 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3642901f-6500-481e-9b26-bb47dde60f46 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:24.715208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:24.725636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3642901f-6500-481e-9b26-bb47dde60f46 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:24.727049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3642901f-6500-481e-9b26-bb47dde60f46 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:24.827348 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 581/1144] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:03:24 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 28 12:03:24.833441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfebfa29-154e-429a-8be2-842334b8de8b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:24.931319 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfebfa29-154e-429a-8be2-842334b8de8b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:24.963707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5eb8d3-e08d-4bae-ac79-f890c81be164 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:24.964338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5eb8d3-e08d-4bae-ac79-f890c81be164 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:24.964639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5eb8d3-e08d-4bae-ac79-f890c81be164 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:24.964909 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5eb8d3-e08d-4bae-ac79-f890c81be164 None admin] PATH_INFO: `/v3/projects/385d211f2dd149f68a2b5a2e936ee368` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:24.966019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5eb8d3-e08d-4bae-ac79-f890c81be164 None admin] RBAC: Authorizing `identity:get_project(project_id=385d211f2dd149f68a2b5a2e936ee368)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:24.967447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5eb8d3-e08d-4bae-ac79-f890c81be164 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:24.969078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 564/1145] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:03:24 2022] GET /identity/v3/projects/385d211f2dd149f68a2b5a2e936ee368 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:24.975866 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3642901f-6500-481e-9b26-bb47dde60f46 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:25.062191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:25.071562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3642901f-6500-481e-9b26-bb47dde60f46 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:25.110822 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82053bdf-6d77-46be-bdf5-7ad14f2a8282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:25.111464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82053bdf-6d77-46be-bdf5-7ad14f2a8282 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:25.111746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82053bdf-6d77-46be-bdf5-7ad14f2a8282 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:25.112019 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82053bdf-6d77-46be-bdf5-7ad14f2a8282 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:25.112780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82053bdf-6d77-46be-bdf5-7ad14f2a8282 None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=385d211f2dd149f68a2b5a2e936ee368, user.id=3748cf2128744fac8c418d98ae6d0d77) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:25.113118 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82053bdf-6d77-46be-bdf5-7ad14f2a8282 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:25.114744 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82053bdf-6d77-46be-bdf5-7ad14f2a8282 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:25.120077 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 582/1146] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:03:24 2022] GET /identity/v3/role_assignments?user.id=3748cf2128744fac8c418d98ae6d0d77&scope.project.id=385d211f2dd149f68a2b5a2e936ee368&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 259 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:27.212583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42314948-9c89-42ae-b392-7d6b67604e42 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:27.213393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42314948-9c89-42ae-b392-7d6b67604e42 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:27.213393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42314948-9c89-42ae-b392-7d6b67604e42 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:27.214714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 565/1147] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:27.219932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f16b736-8943-4cc2-a183-d68a72afd6b5 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:27.220418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f16b736-8943-4cc2-a183-d68a72afd6b5 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:27.220573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f16b736-8943-4cc2-a183-d68a72afd6b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:27.238846 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-5f16b736-8943-4cc2-a183-d68a72afd6b5 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:27.240022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5f16b736-8943-4cc2-a183-d68a72afd6b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:27.304539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 583/1148] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:27.573928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42314948-9c89-42ae-b392-7d6b67604e42 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:27.603305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:27.670628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42314948-9c89-42ae-b392-7d6b67604e42 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:27.702656 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a125be-acff-469c-9197-0580bb51a70a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:27.703254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a125be-acff-469c-9197-0580bb51a70a None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:27.703525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a125be-acff-469c-9197-0580bb51a70a None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:27.703790 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a125be-acff-469c-9197-0580bb51a70a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:27.713616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a125be-acff-469c-9197-0580bb51a70a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-a0a125be-acff-469c-9197-0580bb51a70a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:03:27.718252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:03:27.719147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:03:27.720035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:03:27.720035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:03:27.720035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:03:27.720035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:03:27.720035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:03:27.720035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:03:27.720035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:03:27.720035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:03:27.720035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:03:27.720035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:03:27.720035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:03:27.720035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:03:27.720035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 566/1149] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:03:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:03:27.725799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f16b736-8943-4cc2-a183-d68a72afd6b5 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:27.804945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:27.819022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f16b736-8943-4cc2-a183-d68a72afd6b5 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:27.851033 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bc8d1f-7046-40c9-b861-3f6de61dad15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:27.851616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bc8d1f-7046-40c9-b861-3f6de61dad15 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:27.851890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bc8d1f-7046-40c9-b861-3f6de61dad15 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:27.852159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bc8d1f-7046-40c9-b861-3f6de61dad15 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:27.852845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bc8d1f-7046-40c9-b861-3f6de61dad15 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:27.853143 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bc8d1f-7046-40c9-b861-3f6de61dad15 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:27.854478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bc8d1f-7046-40c9-b861-3f6de61dad15 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:27.862258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 584/1150] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:03:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:27.868024 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a125be-acff-469c-9197-0580bb51a70a None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:27.955346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:27.965085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a125be-acff-469c-9197-0580bb51a70a None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:27.996944 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41367228-4ae6-458c-b0b5-c6c02dc9d478 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:27.997554 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41367228-4ae6-458c-b0b5-c6c02dc9d478 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:27.997829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41367228-4ae6-458c-b0b5-c6c02dc9d478 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:27.998099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41367228-4ae6-458c-b0b5-c6c02dc9d478 None admin] PATH_INFO: `/v3/users/3748cf2128744fac8c418d98ae6d0d77` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:27.999205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41367228-4ae6-458c-b0b5-c6c02dc9d478 None admin] RBAC: Authorizing `identity:get_user(user_id=3748cf2128744fac8c418d98ae6d0d77)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:28.000664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41367228-4ae6-458c-b0b5-c6c02dc9d478 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:28.002255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 567/1151] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:03:27 2022] GET /identity/v3/users/3748cf2128744fac8c418d98ae6d0d77 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:28.008037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bc8d1f-7046-40c9-b861-3f6de61dad15 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:28.105262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bc8d1f-7046-40c9-b861-3f6de61dad15 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:28.137067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f90138-213f-4e4b-b055-09f8d5d4b2e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:28.137675 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f90138-213f-4e4b-b055-09f8d5d4b2e7 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:28.137951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f90138-213f-4e4b-b055-09f8d5d4b2e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:28.138216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f90138-213f-4e4b-b055-09f8d5d4b2e7 None admin] PATH_INFO: `/v3/projects/385d211f2dd149f68a2b5a2e936ee368` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:28.139310 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f90138-213f-4e4b-b055-09f8d5d4b2e7 None admin] RBAC: Authorizing `identity:get_project(project_id=385d211f2dd149f68a2b5a2e936ee368)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:28.140744 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f90138-213f-4e4b-b055-09f8d5d4b2e7 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:28.142874 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 585/1152] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:03:28 2022] GET /identity/v3/projects/385d211f2dd149f68a2b5a2e936ee368 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:28.149015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41367228-4ae6-458c-b0b5-c6c02dc9d478 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:28.243614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41367228-4ae6-458c-b0b5-c6c02dc9d478 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:28.275774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f204de-88c0-4d99-b005-e6b8078429ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:28.276432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f204de-88c0-4d99-b005-e6b8078429ed None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:28.276708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f204de-88c0-4d99-b005-e6b8078429ed None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:28.276977 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f204de-88c0-4d99-b005-e6b8078429ed None admin] PATH_INFO: `/v3/projects/385d211f2dd149f68a2b5a2e936ee368/users/3748cf2128744fac8c418d98ae6d0d77/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:28.278825 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f204de-88c0-4d99-b005-e6b8078429ed None admin] RBAC: Authorizing `identity:create_grant(project_id=385d211f2dd149f68a2b5a2e936ee368, user_id=3748cf2128744fac8c418d98ae6d0d77, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:28.280230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f204de-88c0-4d99-b005-e6b8078429ed None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:28.293270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 568/1153] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:03:28 2022] PUT /identity/v3/projects/385d211f2dd149f68a2b5a2e936ee368/users/3748cf2128744fac8c418d98ae6d0d77/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:03:30.377824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a97b333-4ef0-49b3-a614-e1d078297284 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:30.378390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a97b333-4ef0-49b3-a614-e1d078297284 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:30.378390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a97b333-4ef0-49b3-a614-e1d078297284 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:30.379695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 586/1154] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:30.384834 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6329967d-52c0-4bd1-b814-8397cf5cde2b None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:30.385249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6329967d-52c0-4bd1-b814-8397cf5cde2b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:30.385383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6329967d-52c0-4bd1-b814-8397cf5cde2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:30.403516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-6329967d-52c0-4bd1-b814-8397cf5cde2b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:30.404795 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6329967d-52c0-4bd1-b814-8397cf5cde2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:30.418024 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:30.470753 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 569/1155] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:30.738483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a97b333-4ef0-49b3-a614-e1d078297284 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:30.756441 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:30.836968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a97b333-4ef0-49b3-a614-e1d078297284 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:30.869352 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f0cc25-b659-4daf-af71-3a22358a94c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:30.869958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f0cc25-b659-4daf-af71-3a22358a94c5 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:30.870237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f0cc25-b659-4daf-af71-3a22358a94c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:30.870508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f0cc25-b659-4daf-af71-3a22358a94c5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:30.880871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f0cc25-b659-4daf-af71-3a22358a94c5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-a3f0cc25-b659-4daf-af71-3a22358a94c5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:03:30.885582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:03:30.886633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:03:30.887452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:03:30.887452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:03:30.887452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:03:30.887452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:03:30.887452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:03:30.887452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:03:30.887452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:03:30.887452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:03:30.887452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:03:30.887452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:03:30.887452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:03:30.887452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:03:30.887452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 587/1156] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:03:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:03:30.892784 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6329967d-52c0-4bd1-b814-8397cf5cde2b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:30.986810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6329967d-52c0-4bd1-b814-8397cf5cde2b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:31.019008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d487cbc8-a16c-450c-883c-2bb30eed24a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:31.019535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d487cbc8-a16c-450c-883c-2bb30eed24a7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:31.019813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d487cbc8-a16c-450c-883c-2bb30eed24a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:31.020080 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d487cbc8-a16c-450c-883c-2bb30eed24a7 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:31.020758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d487cbc8-a16c-450c-883c-2bb30eed24a7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:31.021059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d487cbc8-a16c-450c-883c-2bb30eed24a7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:31.022413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d487cbc8-a16c-450c-883c-2bb30eed24a7 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:31.030008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 570/1157] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:03:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:31.036108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f0cc25-b659-4daf-af71-3a22358a94c5 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:31.109547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:31.132075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f0cc25-b659-4daf-af71-3a22358a94c5 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:31.163732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5313e9-d118-4bf8-bc85-42ab4016f0e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:31.164374 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5313e9-d118-4bf8-bc85-42ab4016f0e9 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:31.164655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5313e9-d118-4bf8-bc85-42ab4016f0e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:31.164925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5313e9-d118-4bf8-bc85-42ab4016f0e9 None admin] PATH_INFO: `/v3/users/3748cf2128744fac8c418d98ae6d0d77` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:31.166044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5313e9-d118-4bf8-bc85-42ab4016f0e9 None admin] RBAC: Authorizing `identity:get_user(user_id=3748cf2128744fac8c418d98ae6d0d77)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:31.167456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5313e9-d118-4bf8-bc85-42ab4016f0e9 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:31.169330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 588/1158] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:03:31 2022] GET /identity/v3/users/3748cf2128744fac8c418d98ae6d0d77 => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:31.175647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d487cbc8-a16c-450c-883c-2bb30eed24a7 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:31.187871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:31.273136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d487cbc8-a16c-450c-883c-2bb30eed24a7 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:31.305462 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd138a0-4a49-4e17-b0a8-4119836717a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:31.306054 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd138a0-4a49-4e17-b0a8-4119836717a6 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:31.306330 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd138a0-4a49-4e17-b0a8-4119836717a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:31.306598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd138a0-4a49-4e17-b0a8-4119836717a6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:31.369696 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd138a0-4a49-4e17-b0a8-4119836717a6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:31.371123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd138a0-4a49-4e17-b0a8-4119836717a6 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:31.381969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:31.465125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 571/1159] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:03:31 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:03:31.471135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5313e9-d118-4bf8-bc85-42ab4016f0e9 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:31.566297 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5313e9-d118-4bf8-bc85-42ab4016f0e9 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:31.598131 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf686c1-801b-4003-a3f0-a63cece32e99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:31.598734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf686c1-801b-4003-a3f0-a63cece32e99 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:31.599008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf686c1-801b-4003-a3f0-a63cece32e99 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:31.599279 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf686c1-801b-4003-a3f0-a63cece32e99 None admin] PATH_INFO: `/v3/projects/385d211f2dd149f68a2b5a2e936ee368` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:31.600399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf686c1-801b-4003-a3f0-a63cece32e99 None admin] RBAC: Authorizing `identity:get_project(project_id=385d211f2dd149f68a2b5a2e936ee368)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:31.601797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf686c1-801b-4003-a3f0-a63cece32e99 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:31.603384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 589/1160] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:03:31 2022] GET /identity/v3/projects/385d211f2dd149f68a2b5a2e936ee368 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:03:31.609521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd138a0-4a49-4e17-b0a8-4119836717a6 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:31.703627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd138a0-4a49-4e17-b0a8-4119836717a6 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:31.713869 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:31.736482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8964691-ff5d-4025-8aca-b907a135b69c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:31.737065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8964691-ff5d-4025-8aca-b907a135b69c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:31.737337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8964691-ff5d-4025-8aca-b907a135b69c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:31.737600 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8964691-ff5d-4025-8aca-b907a135b69c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:31.738310 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8964691-ff5d-4025-8aca-b907a135b69c None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=385d211f2dd149f68a2b5a2e936ee368, user.id=3748cf2128744fac8c418d98ae6d0d77) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:31.738613 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8964691-ff5d-4025-8aca-b907a135b69c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:31.740022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8964691-ff5d-4025-8aca-b907a135b69c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:31.745511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 572/1161] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:03:31 2022] GET /identity/v3/role_assignments?user.id=3748cf2128744fac8c418d98ae6d0d77&scope.project.id=385d211f2dd149f68a2b5a2e936ee368&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 608 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:33.877692 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd406287-adb6-4c3f-b626-5abc062c0211 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:33.878271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd406287-adb6-4c3f-b626-5abc062c0211 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:33.878271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd406287-adb6-4c3f-b626-5abc062c0211 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:33.879558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 590/1162] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:33.884695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dbb586-949a-4447-a314-d742623d4c84 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:33.885101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dbb586-949a-4447-a314-d742623d4c84 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:33.885234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dbb586-949a-4447-a314-d742623d4c84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:33.903258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-10dbb586-949a-4447-a314-d742623d4c84 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:33.904459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-10dbb586-949a-4447-a314-d742623d4c84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:33.967694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 573/1163] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:34.234380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd406287-adb6-4c3f-b626-5abc062c0211 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:34.249193 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:34.335315 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd406287-adb6-4c3f-b626-5abc062c0211 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:34.367083 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62829539-2999-4a69-86b9-5fe5aff592bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:34.367676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62829539-2999-4a69-86b9-5fe5aff592bd None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:34.367956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62829539-2999-4a69-86b9-5fe5aff592bd None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:34.368226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62829539-2999-4a69-86b9-5fe5aff592bd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:34.432715 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62829539-2999-4a69-86b9-5fe5aff592bd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:34.434152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62829539-2999-4a69-86b9-5fe5aff592bd None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:34.446673 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:34.527890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 591/1164] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:03:34 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:03:34.534361 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dbb586-949a-4447-a314-d742623d4c84 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:34.628639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dbb586-949a-4447-a314-d742623d4c84 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:34.630849 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:34.662910 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb69e84-670e-4373-8fc6-08208d5ff3e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:34.663522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb69e84-670e-4373-8fc6-08208d5ff3e0 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:34.663799 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb69e84-670e-4373-8fc6-08208d5ff3e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:34.664067 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb69e84-670e-4373-8fc6-08208d5ff3e0 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:34.670496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb69e84-670e-4373-8fc6-08208d5ff3e0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:34.671931 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb69e84-670e-4373-8fc6-08208d5ff3e0 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-adb69e84-670e-4373-8fc6-08208d5ff3e0 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:03:34.682870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:03:34.684171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:03:34.685268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 574/1165] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Sep 28 12:03:34 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:03:34.689844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62829539-2999-4a69-86b9-5fe5aff592bd None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:34.784324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62829539-2999-4a69-86b9-5fe5aff592bd None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:34.796628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:34.817196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b0de4d-8098-4a7f-8371-9687aa1efef2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:34.817788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b0de4d-8098-4a7f-8371-9687aa1efef2 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:34.818065 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b0de4d-8098-4a7f-8371-9687aa1efef2 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:34.818335 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b0de4d-8098-4a7f-8371-9687aa1efef2 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:34.818972 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b0de4d-8098-4a7f-8371-9687aa1efef2 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:34.819273 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b0de4d-8098-4a7f-8371-9687aa1efef2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:34.820640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b0de4d-8098-4a7f-8371-9687aa1efef2 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:34.827384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 592/1166] 10.222.0.26 () {56 vars in 1115 bytes} [Wed Sep 28 12:03:34 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:03:36.944481 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f50ffb4-6c4f-4688-8e9b-468c7af42dd7 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:36.945050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f50ffb4-6c4f-4688-8e9b-468c7af42dd7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:36.945050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f50ffb4-6c4f-4688-8e9b-468c7af42dd7 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:36.946356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 575/1167] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:36.951568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f129e6f4-228c-4105-bc3b-696caa33f9df None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:36.951949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f129e6f4-228c-4105-bc3b-696caa33f9df None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:36.952112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f129e6f4-228c-4105-bc3b-696caa33f9df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:36.970992 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-f129e6f4-228c-4105-bc3b-696caa33f9df None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:36.972153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f129e6f4-228c-4105-bc3b-696caa33f9df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:37.036818 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 593/1168] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:37.302483 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f50ffb4-6c4f-4688-8e9b-468c7af42dd7 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:37.401862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f50ffb4-6c4f-4688-8e9b-468c7af42dd7 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:37.434861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948626f5-b5e5-4b4e-bdcc-8e11678b84fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:37.435450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948626f5-b5e5-4b4e-bdcc-8e11678b84fe None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:37.435727 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948626f5-b5e5-4b4e-bdcc-8e11678b84fe None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:37.435999 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948626f5-b5e5-4b4e-bdcc-8e11678b84fe None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:37.436696 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948626f5-b5e5-4b4e-bdcc-8e11678b84fe None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:37.438057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948626f5-b5e5-4b4e-bdcc-8e11678b84fe None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:37.453444 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 576/1169] 10.222.0.26 () {60 vars in 1138 bytes} [Wed Sep 28 12:03:37 2022] POST /identity/v3/domains => generated 267 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:03:39.599010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d78f18-133f-450c-8e71-95011dfd16b5 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:39.599557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d78f18-133f-450c-8e71-95011dfd16b5 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:39.599557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d78f18-133f-450c-8e71-95011dfd16b5 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:39.600898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 594/1170] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:39.606293 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29d8c74-1eaf-4169-85f0-4a9349b96454 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:39.606678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29d8c74-1eaf-4169-85f0-4a9349b96454 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:39.606846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29d8c74-1eaf-4169-85f0-4a9349b96454 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:39.624539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-e29d8c74-1eaf-4169-85f0-4a9349b96454 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:39.625717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e29d8c74-1eaf-4169-85f0-4a9349b96454 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:39.643352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:39.690895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 577/1171] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:39.963440 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d78f18-133f-450c-8e71-95011dfd16b5 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:40.060185 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d78f18-133f-450c-8e71-95011dfd16b5 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:40.068757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:40.094123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b828d3a4-e9ac-4247-9654-69afb5a3d292 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:40.094736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b828d3a4-e9ac-4247-9654-69afb5a3d292 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:40.095008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b828d3a4-e9ac-4247-9654-69afb5a3d292 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:40.095274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b828d3a4-e9ac-4247-9654-69afb5a3d292 None admin] PATH_INFO: `/v3/domains/7edcd31a93584ac8beff07bf1128f676` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:40.105076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b828d3a4-e9ac-4247-9654-69afb5a3d292 None admin] RBAC: Authorizing `identity:get_domain(domain_id=7edcd31a93584ac8beff07bf1128f676)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:40.106483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b828d3a4-e9ac-4247-9654-69afb5a3d292 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:40.108028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 595/1172] 10.222.0.26 () {56 vars in 1183 bytes} [Wed Sep 28 12:03:39 2022] GET /identity/v3/domains/7edcd31a93584ac8beff07bf1128f676 => generated 267 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:40.114380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29d8c74-1eaf-4169-85f0-4a9349b96454 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:40.219067 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29d8c74-1eaf-4169-85f0-4a9349b96454 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:40.251135 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad026b42-ad6a-45a6-b0f5-0ee2fdc1af46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:40.252259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad026b42-ad6a-45a6-b0f5-0ee2fdc1af46 None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:40.252569 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad026b42-ad6a-45a6-b0f5-0ee2fdc1af46 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:40.252839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad026b42-ad6a-45a6-b0f5-0ee2fdc1af46 None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:40.253491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad026b42-ad6a-45a6-b0f5-0ee2fdc1af46 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:40.254923 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad026b42-ad6a-45a6-b0f5-0ee2fdc1af46 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:40.279741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 578/1173] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Sep 28 12:03:40 2022] POST /identity/v3/projects => generated 361 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:03:42.446344 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffb0771-4418-48fc-907e-586f5951c915 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:42.446878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffb0771-4418-48fc-907e-586f5951c915 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:42.446878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffb0771-4418-48fc-907e-586f5951c915 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:42.448211 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 596/1174] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:42.453660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baacdbfa-778a-4557-8098-f89c7ed4a8fc None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:42.454064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baacdbfa-778a-4557-8098-f89c7ed4a8fc None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:42.454699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baacdbfa-778a-4557-8098-f89c7ed4a8fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:42.467541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:42.473653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-baacdbfa-778a-4557-8098-f89c7ed4a8fc None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:42.474953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-baacdbfa-778a-4557-8098-f89c7ed4a8fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:42.538726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 579/1175] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:42.804502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffb0771-4418-48fc-907e-586f5951c915 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:42.901372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffb0771-4418-48fc-907e-586f5951c915 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:42.933278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c335185-c332-4d0d-9587-8674f47ecfb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:42.933882 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c335185-c332-4d0d-9587-8674f47ecfb5 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:42.934160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c335185-c332-4d0d-9587-8674f47ecfb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:42.934431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c335185-c332-4d0d-9587-8674f47ecfb5 None admin] PATH_INFO: `/v3/domains/7edcd31a93584ac8beff07bf1128f676` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:42.935534 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c335185-c332-4d0d-9587-8674f47ecfb5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=7edcd31a93584ac8beff07bf1128f676)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:42.936964 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c335185-c332-4d0d-9587-8674f47ecfb5 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:42.938476 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 597/1176] 10.222.0.26 () {56 vars in 1183 bytes} [Wed Sep 28 12:03:42 2022] GET /identity/v3/domains/7edcd31a93584ac8beff07bf1128f676 => generated 267 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:42.944052 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baacdbfa-778a-4557-8098-f89c7ed4a8fc None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:43.039399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baacdbfa-778a-4557-8098-f89c7ed4a8fc None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:43.071294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052035b8-b302-4902-882a-efa192d62406 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:43.071829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052035b8-b302-4902-882a-efa192d62406 None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:43.072108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052035b8-b302-4902-882a-efa192d62406 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:43.072409 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052035b8-b302-4902-882a-efa192d62406 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:43.073051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052035b8-b302-4902-882a-efa192d62406 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:43.074475 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052035b8-b302-4902-882a-efa192d62406 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:43.083733 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:43.097366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 580/1177] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:03:42 2022] POST /identity/v3/users => generated 311 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:03:45.256015 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a918be60-601b-4b9b-9cc6-2861a2c02263 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:45.256571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a918be60-601b-4b9b-9cc6-2861a2c02263 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:45.256571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a918be60-601b-4b9b-9cc6-2861a2c02263 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:45.257945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 598/1178] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:45.263170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc0c185-bbd1-4812-b0fd-b9b5f33118e7 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:45.263578 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc0c185-bbd1-4812-b0fd-b9b5f33118e7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:45.263716 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc0c185-bbd1-4812-b0fd-b9b5f33118e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:45.283220 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-8cc0c185-bbd1-4812-b0fd-b9b5f33118e7 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:45.284415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8cc0c185-bbd1-4812-b0fd-b9b5f33118e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:45.347616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 581/1179] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:45.615026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a918be60-601b-4b9b-9cc6-2861a2c02263 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:45.648570 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:45.712892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a918be60-601b-4b9b-9cc6-2861a2c02263 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:45.752781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ec5664-358d-43bc-803f-f46d593e39cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:45.753394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ec5664-358d-43bc-803f-f46d593e39cd None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:45.753669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ec5664-358d-43bc-803f-f46d593e39cd None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:45.753945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ec5664-358d-43bc-803f-f46d593e39cd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:45.763554 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ec5664-358d-43bc-803f-f46d593e39cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-36ec5664-358d-43bc-803f-f46d593e39cd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:03:45.768253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:03:45.769349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:03:45.770396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:03:45.770396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:03:45.770396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:03:45.770396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:03:45.770396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:03:45.770396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:03:45.770396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:03:45.770396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:03:45.770396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:03:45.770396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:03:45.770396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:03:45.770396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:03:45.770396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 599/1180] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:03:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:03:45.775307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc0c185-bbd1-4812-b0fd-b9b5f33118e7 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:45.869365 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc0c185-bbd1-4812-b0fd-b9b5f33118e7 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:45.890164 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:45.902796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b548ed87-49c9-42cc-bcbe-8193b8e35422 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:45.903381 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b548ed87-49c9-42cc-bcbe-8193b8e35422 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:45.903656 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b548ed87-49c9-42cc-bcbe-8193b8e35422 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:45.903923 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b548ed87-49c9-42cc-bcbe-8193b8e35422 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:45.904612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b548ed87-49c9-42cc-bcbe-8193b8e35422 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:45.904911 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b548ed87-49c9-42cc-bcbe-8193b8e35422 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:45.906258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b548ed87-49c9-42cc-bcbe-8193b8e35422 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:45.914414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 582/1181] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:03:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:45.920174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ec5664-358d-43bc-803f-f46d593e39cd None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:46.010216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:46.017021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ec5664-358d-43bc-803f-f46d593e39cd None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:46.054693 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3c18a2-e3ce-4258-b610-9540bc09aedf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:46.055586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3c18a2-e3ce-4258-b610-9540bc09aedf None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:46.056026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3c18a2-e3ce-4258-b610-9540bc09aedf None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:46.056510 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3c18a2-e3ce-4258-b610-9540bc09aedf None admin] PATH_INFO: `/v3/users/9062aa07a02d4cb2b3037b9eb64c60c8` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:46.071027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3c18a2-e3ce-4258-b610-9540bc09aedf None admin] RBAC: Authorizing `identity:get_user(user_id=9062aa07a02d4cb2b3037b9eb64c60c8)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:46.072544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3c18a2-e3ce-4258-b610-9540bc09aedf None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:46.074154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 600/1182] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:03:45 2022] GET /identity/v3/users/9062aa07a02d4cb2b3037b9eb64c60c8 => generated 311 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:46.079969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b548ed87-49c9-42cc-bcbe-8193b8e35422 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:46.177854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b548ed87-49c9-42cc-bcbe-8193b8e35422 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:46.210642 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49eaea38-59f4-4dad-8bd1-c620ca739292 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:46.211230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eaea38-59f4-4dad-8bd1-c620ca739292 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:46.211506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eaea38-59f4-4dad-8bd1-c620ca739292 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:46.211774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eaea38-59f4-4dad-8bd1-c620ca739292 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:46.260783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:46.275953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49eaea38-59f4-4dad-8bd1-c620ca739292 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:46.277427 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49eaea38-59f4-4dad-8bd1-c620ca739292 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:46.369634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 583/1183] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:03:46 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:03:46.375596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3c18a2-e3ce-4258-b610-9540bc09aedf None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:46.471180 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3c18a2-e3ce-4258-b610-9540bc09aedf None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:46.507351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e495aacc-dfb0-4fca-b034-244b2c48d991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:46.507966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e495aacc-dfb0-4fca-b034-244b2c48d991 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:46.508245 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e495aacc-dfb0-4fca-b034-244b2c48d991 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:46.508544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e495aacc-dfb0-4fca-b034-244b2c48d991 None admin] PATH_INFO: `/v3/projects/1a420aa8c58e42bfa42c2f4fd85bff76` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:46.509646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e495aacc-dfb0-4fca-b034-244b2c48d991 None admin] RBAC: Authorizing `identity:get_project(project_id=1a420aa8c58e42bfa42c2f4fd85bff76)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:46.511047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e495aacc-dfb0-4fca-b034-244b2c48d991 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:46.512680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 601/1184] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:03:46 2022] GET /identity/v3/projects/1a420aa8c58e42bfa42c2f4fd85bff76 => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:46.519282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49eaea38-59f4-4dad-8bd1-c620ca739292 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:46.606300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:46.622565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49eaea38-59f4-4dad-8bd1-c620ca739292 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:46.654752 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b110f0-5cbe-49c6-94d2-d8870ecb27fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:46.655343 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b110f0-5cbe-49c6-94d2-d8870ecb27fc None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:46.655621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b110f0-5cbe-49c6-94d2-d8870ecb27fc None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:46.655892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b110f0-5cbe-49c6-94d2-d8870ecb27fc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:46.656641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b110f0-5cbe-49c6-94d2-d8870ecb27fc None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=1a420aa8c58e42bfa42c2f4fd85bff76, user.id=9062aa07a02d4cb2b3037b9eb64c60c8) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:46.656945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b110f0-5cbe-49c6-94d2-d8870ecb27fc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:46.658393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b110f0-5cbe-49c6-94d2-d8870ecb27fc None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:46.663499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 584/1185] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:03:46 2022] GET /identity/v3/role_assignments?user.id=9062aa07a02d4cb2b3037b9eb64c60c8&scope.project.id=1a420aa8c58e42bfa42c2f4fd85bff76&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 259 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:48.769729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961a98cd-91b4-4a6f-af17-dee57eee914c None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:48.770237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961a98cd-91b4-4a6f-af17-dee57eee914c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:48.770237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961a98cd-91b4-4a6f-af17-dee57eee914c None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:48.771507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 602/1186] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:48.776711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aa42cf-d61a-4892-85da-a073484db48f None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:48.777109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aa42cf-d61a-4892-85da-a073484db48f None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:48.777261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aa42cf-d61a-4892-85da-a073484db48f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:48.795407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-b8aa42cf-d61a-4892-85da-a073484db48f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:48.796626 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b8aa42cf-d61a-4892-85da-a073484db48f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:48.861896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 585/1187] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:49.130116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961a98cd-91b4-4a6f-af17-dee57eee914c None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:49.151393 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:49.226875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961a98cd-91b4-4a6f-af17-dee57eee914c None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:49.259213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b87d828-dfbf-4075-a8a9-4cd7331b2873 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:49.259817 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b87d828-dfbf-4075-a8a9-4cd7331b2873 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:49.260092 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b87d828-dfbf-4075-a8a9-4cd7331b2873 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:49.260384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b87d828-dfbf-4075-a8a9-4cd7331b2873 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:49.269858 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b87d828-dfbf-4075-a8a9-4cd7331b2873 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-5b87d828-dfbf-4075-a8a9-4cd7331b2873 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:03:49.274487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:03:49.275556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:03:49.276612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:03:49.276612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:03:49.276612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:03:49.276612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:03:49.276612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:03:49.276612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:03:49.276612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:03:49.276612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:03:49.276612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:03:49.276612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:03:49.276612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:03:49.276612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:03:49.276612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 603/1188] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:03:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:03:49.283751 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8aa42cf-d61a-4892-85da-a073484db48f None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:49.357430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:49.378672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8aa42cf-d61a-4892-85da-a073484db48f None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:49.410842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c74dbc-87d8-4011-9c50-1a04e6e20eed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:49.411375 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c74dbc-87d8-4011-9c50-1a04e6e20eed None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:49.411652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c74dbc-87d8-4011-9c50-1a04e6e20eed None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:49.411921 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c74dbc-87d8-4011-9c50-1a04e6e20eed None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:49.412603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c74dbc-87d8-4011-9c50-1a04e6e20eed None admin] RBAC: Adding query filter params (name=admin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:49.412900 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c74dbc-87d8-4011-9c50-1a04e6e20eed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:49.414251 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c74dbc-87d8-4011-9c50-1a04e6e20eed None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:49.421907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 586/1189] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:03:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:49.428360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b87d828-dfbf-4075-a8a9-4cd7331b2873 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:49.505523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:49.523911 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b87d828-dfbf-4075-a8a9-4cd7331b2873 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:49.558021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7b866c-85a9-4b6c-a941-ba1e419197cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:49.558636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b866c-85a9-4b6c-a941-ba1e419197cc None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:49.558913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b866c-85a9-4b6c-a941-ba1e419197cc None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:49.559182 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b866c-85a9-4b6c-a941-ba1e419197cc None admin] PATH_INFO: `/v3/users/9062aa07a02d4cb2b3037b9eb64c60c8` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:49.560291 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7b866c-85a9-4b6c-a941-ba1e419197cc None admin] RBAC: Authorizing `identity:get_user(user_id=9062aa07a02d4cb2b3037b9eb64c60c8)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:49.561736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7b866c-85a9-4b6c-a941-ba1e419197cc None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:49.563278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 604/1190] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:03:49 2022] GET /identity/v3/users/9062aa07a02d4cb2b3037b9eb64c60c8 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:49.569014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c74dbc-87d8-4011-9c50-1a04e6e20eed None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:49.663866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c74dbc-87d8-4011-9c50-1a04e6e20eed None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:49.694872 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c164d93-9fc8-4c50-9b32-d2971e1e79a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:49.695475 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c164d93-9fc8-4c50-9b32-d2971e1e79a2 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:49.695751 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c164d93-9fc8-4c50-9b32-d2971e1e79a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:49.696019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c164d93-9fc8-4c50-9b32-d2971e1e79a2 None admin] PATH_INFO: `/v3/projects/1a420aa8c58e42bfa42c2f4fd85bff76` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:49.697151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c164d93-9fc8-4c50-9b32-d2971e1e79a2 None admin] RBAC: Authorizing `identity:get_project(project_id=1a420aa8c58e42bfa42c2f4fd85bff76)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:49.698554 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c164d93-9fc8-4c50-9b32-d2971e1e79a2 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:49.700117 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 587/1191] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:03:49 2022] GET /identity/v3/projects/1a420aa8c58e42bfa42c2f4fd85bff76 => generated 361 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:49.706302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7b866c-85a9-4b6c-a941-ba1e419197cc None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:49.801307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7b866c-85a9-4b6c-a941-ba1e419197cc None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:49.833353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920809b3-ddf9-4646-bf50-66bd60b30981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:49.833983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920809b3-ddf9-4646-bf50-66bd60b30981 None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:49.834261 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920809b3-ddf9-4646-bf50-66bd60b30981 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:49.834533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920809b3-ddf9-4646-bf50-66bd60b30981 None admin] PATH_INFO: `/v3/projects/1a420aa8c58e42bfa42c2f4fd85bff76/users/9062aa07a02d4cb2b3037b9eb64c60c8/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:49.836424 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920809b3-ddf9-4646-bf50-66bd60b30981 None admin] RBAC: Authorizing `identity:create_grant(project_id=1a420aa8c58e42bfa42c2f4fd85bff76, user_id=9062aa07a02d4cb2b3037b9eb64c60c8, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:49.837817 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920809b3-ddf9-4646-bf50-66bd60b30981 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:49.850192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 605/1192] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:03:49 2022] PUT /identity/v3/projects/1a420aa8c58e42bfa42c2f4fd85bff76/users/9062aa07a02d4cb2b3037b9eb64c60c8/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:03:51.947045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff49f38c-2d50-4f03-adc5-4704b3c003a7 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:51.947581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff49f38c-2d50-4f03-adc5-4704b3c003a7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:51.947581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff49f38c-2d50-4f03-adc5-4704b3c003a7 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:51.948948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 588/1193] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:03:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:03:51.954756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3acfa4-88d5-4eb1-b59f-8185c66dd12d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:51.955149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3acfa4-88d5-4eb1-b59f-8185c66dd12d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:51.955301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3acfa4-88d5-4eb1-b59f-8185c66dd12d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:51.973808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-da3acfa4-88d5-4eb1-b59f-8185c66dd12d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:03:51.974973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-da3acfa4-88d5-4eb1-b59f-8185c66dd12d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:03:51.976916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:52.040173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 606/1194] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:03:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:03:52.308006 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff49f38c-2d50-4f03-adc5-4704b3c003a7 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:52.319198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:52.404662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff49f38c-2d50-4f03-adc5-4704b3c003a7 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:52.436714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7f6274-b38e-4d00-ae3e-3c73dd8ab597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:52.437316 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7f6274-b38e-4d00-ae3e-3c73dd8ab597 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:52.437597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7f6274-b38e-4d00-ae3e-3c73dd8ab597 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:52.437865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7f6274-b38e-4d00-ae3e-3c73dd8ab597 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:52.448641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7f6274-b38e-4d00-ae3e-3c73dd8ab597 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-eb7f6274-b38e-4d00-ae3e-3c73dd8ab597 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:03:52.453263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:03:52.454402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:03:52.455522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:03:52.455522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:03:52.455522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:03:52.455522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:03:52.455522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:03:52.455522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:03:52.455522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:03:52.455522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:03:52.455522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:03:52.455522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:03:52.455522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:03:52.455522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:03:52.455522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 589/1195] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:03:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:03:52.460537 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3acfa4-88d5-4eb1-b59f-8185c66dd12d None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:52.555524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3acfa4-88d5-4eb1-b59f-8185c66dd12d None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:52.587129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe548d5-2c88-4631-9408-7218239739f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:52.587660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe548d5-2c88-4631-9408-7218239739f5 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:52.588024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe548d5-2c88-4631-9408-7218239739f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:52.588577 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe548d5-2c88-4631-9408-7218239739f5 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:52.589241 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe548d5-2c88-4631-9408-7218239739f5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:52.589542 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe548d5-2c88-4631-9408-7218239739f5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:52.590876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe548d5-2c88-4631-9408-7218239739f5 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:52.596151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:52.599769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 607/1196] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:03:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:52.605436 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7f6274-b38e-4d00-ae3e-3c73dd8ab597 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:52.668960 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:52.700399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7f6274-b38e-4d00-ae3e-3c73dd8ab597 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:52.732779 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fce3a2e-09d2-4a8b-9ad8-b03256046aae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:52.733382 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce3a2e-09d2-4a8b-9ad8-b03256046aae None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:52.733659 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce3a2e-09d2-4a8b-9ad8-b03256046aae None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:52.733926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce3a2e-09d2-4a8b-9ad8-b03256046aae None admin] PATH_INFO: `/v3/users/9062aa07a02d4cb2b3037b9eb64c60c8` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:52.735023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fce3a2e-09d2-4a8b-9ad8-b03256046aae None admin] RBAC: Authorizing `identity:get_user(user_id=9062aa07a02d4cb2b3037b9eb64c60c8)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:52.736499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fce3a2e-09d2-4a8b-9ad8-b03256046aae None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:52.738023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 590/1197] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Sep 28 12:03:52 2022] GET /identity/v3/users/9062aa07a02d4cb2b3037b9eb64c60c8 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:52.743774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe548d5-2c88-4631-9408-7218239739f5 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:52.837845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe548d5-2c88-4631-9408-7218239739f5 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:52.869327 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ecef3e-e05c-4be1-963c-6e08c68b2190 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:52.869908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ecef3e-e05c-4be1-963c-6e08c68b2190 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:52.870182 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ecef3e-e05c-4be1-963c-6e08c68b2190 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:52.870448 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ecef3e-e05c-4be1-963c-6e08c68b2190 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:52.932224 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:52.934780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ecef3e-e05c-4be1-963c-6e08c68b2190 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:52.936690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ecef3e-e05c-4be1-963c-6e08c68b2190 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:53.030482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 608/1198] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:03:52 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 287 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:03:53.037168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fce3a2e-09d2-4a8b-9ad8-b03256046aae None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:53.131629 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fce3a2e-09d2-4a8b-9ad8-b03256046aae None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:53.163074 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a24b8b1-bae0-4ef6-af88-675c6b5e1db4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:53.163678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24b8b1-bae0-4ef6-af88-675c6b5e1db4 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:53.163955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24b8b1-bae0-4ef6-af88-675c6b5e1db4 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:53.164225 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24b8b1-bae0-4ef6-af88-675c6b5e1db4 None admin] PATH_INFO: `/v3/projects/1a420aa8c58e42bfa42c2f4fd85bff76` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:53.165351 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a24b8b1-bae0-4ef6-af88-675c6b5e1db4 None admin] RBAC: Authorizing `identity:get_project(project_id=1a420aa8c58e42bfa42c2f4fd85bff76)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:53.166765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a24b8b1-bae0-4ef6-af88-675c6b5e1db4 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:53.168355 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 591/1199] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Sep 28 12:03:53 2022] GET /identity/v3/projects/1a420aa8c58e42bfa42c2f4fd85bff76 => generated 361 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:03:53.174802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ecef3e-e05c-4be1-963c-6e08c68b2190 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:03:53.266272 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:03:53.269323 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ecef3e-e05c-4be1-963c-6e08c68b2190 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:03:53.316341 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069b35ff-8535-4998-99ae-951a82e5b5ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:03:53.316960 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b35ff-8535-4998-99ae-951a82e5b5ea None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:03:53.317233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b35ff-8535-4998-99ae-951a82e5b5ea None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:03:53.317499 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b35ff-8535-4998-99ae-951a82e5b5ea None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:03:53.318211 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069b35ff-8535-4998-99ae-951a82e5b5ea None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=1a420aa8c58e42bfa42c2f4fd85bff76, user.id=9062aa07a02d4cb2b3037b9eb64c60c8) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:03:53.318508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069b35ff-8535-4998-99ae-951a82e5b5ea None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:03:53.319905 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069b35ff-8535-4998-99ae-951a82e5b5ea None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:03:53.325488 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 609/1200] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:03:53 2022] GET /identity/v3/role_assignments?user.id=9062aa07a02d4cb2b3037b9eb64c60c8&scope.project.id=1a420aa8c58e42bfa42c2f4fd85bff76&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:04:03.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=98 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:03.322945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=18 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:49.161783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c0a1f-adf2-421e-a78b-bea9d9b827b4 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:49.162398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c0a1f-adf2-421e-a78b-bea9d9b827b4 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:49.162398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c0a1f-adf2-421e-a78b-bea9d9b827b4 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:49.163799 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 592/1201] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:04:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:04:49.170817 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6eccdeb-643c-40cf-9064-134d89b68ae8 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:49.171358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6eccdeb-643c-40cf-9064-134d89b68ae8 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:49.171358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6eccdeb-643c-40cf-9064-134d89b68ae8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:49.193779 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-e6eccdeb-643c-40cf-9064-134d89b68ae8 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:04:49.195280 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e6eccdeb-643c-40cf-9064-134d89b68ae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:04:49.287501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 610/1202] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:04:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:04:49.561727 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9c0a1f-adf2-421e-a78b-bea9d9b827b4 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:49.693630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9c0a1f-adf2-421e-a78b-bea9d9b827b4 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:49.731291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f87f78-18c4-4da4-8b2a-9acd4befcb87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:49.731903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f87f78-18c4-4da4-8b2a-9acd4befcb87 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:49.732237 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f87f78-18c4-4da4-8b2a-9acd4befcb87 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:49.732475 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f87f78-18c4-4da4-8b2a-9acd4befcb87 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:49.743726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f87f78-18c4-4da4-8b2a-9acd4befcb87 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:49.745278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f87f78-18c4-4da4-8b2a-9acd4befcb87 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-56f87f78-18c4-4da4-8b2a-9acd4befcb87 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:04:49.759782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:04:49.760949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:04:49.761937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:04:49.761937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:04:49.761937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:04:49.761937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 593/1203] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:04:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:04:49.767808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6eccdeb-643c-40cf-9064-134d89b68ae8 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:49.899405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6eccdeb-643c-40cf-9064-134d89b68ae8 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:49.938660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:49.945467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798bd9da-c90b-4456-b3c5-c739d3228e97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:49.946150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798bd9da-c90b-4456-b3c5-c739d3228e97 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:49.946481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798bd9da-c90b-4456-b3c5-c739d3228e97 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:49.946797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798bd9da-c90b-4456-b3c5-c739d3228e97 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:49.947511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798bd9da-c90b-4456-b3c5-c739d3228e97 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:04:49.947976 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798bd9da-c90b-4456-b3c5-c739d3228e97 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:49.949683 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798bd9da-c90b-4456-b3c5-c739d3228e97 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:49.966717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 611/1204] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:04:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:04:49.968013 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f87f78-18c4-4da4-8b2a-9acd4befcb87 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:50.028070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:50.084529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f87f78-18c4-4da4-8b2a-9acd4befcb87 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:50.124873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7e85b1-e588-4e40-9180-318d245f8e60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:50.125634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7e85b1-e588-4e40-9180-318d245f8e60 None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:50.126028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7e85b1-e588-4e40-9180-318d245f8e60 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:50.126376 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7e85b1-e588-4e40-9180-318d245f8e60 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:50.127150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7e85b1-e588-4e40-9180-318d245f8e60 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:50.128777 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7e85b1-e588-4e40-9180-318d245f8e60 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:50.168331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 594/1205] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Sep 28 12:04:49 2022] POST /identity/v3/users => generated 251 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:04:52.605698 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b4894d-1e94-4f3a-ab62-f094084395ef None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:52.606241 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b4894d-1e94-4f3a-ab62-f094084395ef None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:52.606241 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b4894d-1e94-4f3a-ab62-f094084395ef None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:52.607770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 612/1206] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:04:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:04:52.614467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8576767d-455e-43b3-9d20-d8f32f1cdf12 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:52.615800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8576767d-455e-43b3-9d20-d8f32f1cdf12 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:52.615800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8576767d-455e-43b3-9d20-d8f32f1cdf12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:52.636579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-8576767d-455e-43b3-9d20-d8f32f1cdf12 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:04:52.637981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8576767d-455e-43b3-9d20-d8f32f1cdf12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:04:52.725388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 595/1207] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:04:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:04:52.990824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b4894d-1e94-4f3a-ab62-f094084395ef None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:53.109500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b4894d-1e94-4f3a-ab62-f094084395ef None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:53.148388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f348ff-fc75-407e-8e4b-7500ee9483a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:53.149050 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f348ff-fc75-407e-8e4b-7500ee9483a2 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:53.149397 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f348ff-fc75-407e-8e4b-7500ee9483a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:53.149764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f348ff-fc75-407e-8e4b-7500ee9483a2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:53.162165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f348ff-fc75-407e-8e4b-7500ee9483a2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-74f348ff-fc75-407e-8e4b-7500ee9483a2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:04:53.167174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:04:53.168322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:04:53.169332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:04:53.169332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:04:53.169332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:04:53.169332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:04:53.169332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:04:53.169332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:04:53.169332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:04:53.169332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:04:53.169332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:04:53.169332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:04:53.169332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:04:53.169332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:04:53.169332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 613/1208] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:04:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:04:53.175089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8576767d-455e-43b3-9d20-d8f32f1cdf12 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:53.223830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:53.305510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8576767d-455e-43b3-9d20-d8f32f1cdf12 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:53.346627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68408e3b-d737-4f0d-aba9-1f39e94932af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:53.347508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68408e3b-d737-4f0d-aba9-1f39e94932af None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:53.347953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68408e3b-d737-4f0d-aba9-1f39e94932af None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:53.348530 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68408e3b-d737-4f0d-aba9-1f39e94932af None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:53.349175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68408e3b-d737-4f0d-aba9-1f39e94932af None admin] RBAC: Adding query filter params (name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:04:53.349472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68408e3b-d737-4f0d-aba9-1f39e94932af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:53.351040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68408e3b-d737-4f0d-aba9-1f39e94932af None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:53.364479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 596/1209] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:04:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:04:53.366467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f348ff-fc75-407e-8e4b-7500ee9483a2 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:53.425962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:53.487327 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f348ff-fc75-407e-8e4b-7500ee9483a2 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:53.527027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc0bc7-af3c-453f-a0a8-c2541020aef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:53.527771 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc0bc7-af3c-453f-a0a8-c2541020aef0 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:53.528103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc0bc7-af3c-453f-a0a8-c2541020aef0 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:53.528447 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc0bc7-af3c-453f-a0a8-c2541020aef0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:53.540208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbc0bc7-af3c-453f-a0a8-c2541020aef0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:53.541774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbc0bc7-af3c-453f-a0a8-c2541020aef0 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-1cbc0bc7-af3c-453f-a0a8-c2541020aef0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:04:53.558462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:04:53.560004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:04:53.561200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:04:53.561200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:04:53.561200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:04:53.561200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 614/1210] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:04:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:04:53.565650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68408e3b-d737-4f0d-aba9-1f39e94932af None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:53.682794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:53.686689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68408e3b-d737-4f0d-aba9-1f39e94932af None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:53.733500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df61e92e-822b-4542-a293-34dafaf12a23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:53.734648 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df61e92e-822b-4542-a293-34dafaf12a23 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:53.734948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df61e92e-822b-4542-a293-34dafaf12a23 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:53.735219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df61e92e-822b-4542-a293-34dafaf12a23 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:53.735860 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df61e92e-822b-4542-a293-34dafaf12a23 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:04:53.736163 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df61e92e-822b-4542-a293-34dafaf12a23 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:53.737558 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df61e92e-822b-4542-a293-34dafaf12a23 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:53.749452 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 597/1211] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:04:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:04:53.755602 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc0bc7-af3c-453f-a0a8-c2541020aef0 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:53.866847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc0bc7-af3c-453f-a0a8-c2541020aef0 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:53.868857 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:53.910154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41abe734-d8cf-49a7-acb0-ef3101617cc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:53.910843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41abe734-d8cf-49a7-acb0-ef3101617cc8 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:53.911120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41abe734-d8cf-49a7-acb0-ef3101617cc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:53.911433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41abe734-d8cf-49a7-acb0-ef3101617cc8 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:53.919895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41abe734-d8cf-49a7-acb0-ef3101617cc8 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:53.921581 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41abe734-d8cf-49a7-acb0-ef3101617cc8 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-41abe734-d8cf-49a7-acb0-ef3101617cc8 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:04:53.934803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:04:53.935889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:04:53.936885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:04:53.936885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:04:53.936885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:04:53.936885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:04:53.936885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:04:53.936885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:04:53.936885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:04:53.936885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:04:53.936885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 28 12:04:53.936885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:04:53.938769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 615/1212] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:04:53 2022] GET /identity/v3/users/placement => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:04:53.944403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df61e92e-822b-4542-a293-34dafaf12a23 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:54.065761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df61e92e-822b-4542-a293-34dafaf12a23 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:54.102687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed27e35-134f-4b5a-abb5-77ecc86696fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:54.103653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed27e35-134f-4b5a-abb5-77ecc86696fd None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:54.104122 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed27e35-134f-4b5a-abb5-77ecc86696fd None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:54.104663 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed27e35-134f-4b5a-abb5-77ecc86696fd None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:54.105573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed27e35-134f-4b5a-abb5-77ecc86696fd None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:04:54.106060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed27e35-134f-4b5a-abb5-77ecc86696fd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:54.107780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed27e35-134f-4b5a-abb5-77ecc86696fd None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:54.129728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 598/1213] 10.222.0.26 () {56 vars in 1143 bytes} [Wed Sep 28 12:04:53 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:04:54.136441 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41abe734-d8cf-49a7-acb0-ef3101617cc8 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:54.279888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41abe734-d8cf-49a7-acb0-ef3101617cc8 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:54.333678 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af72e56-c2e5-4eb9-a9c9-1d236618c9fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:54.334291 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af72e56-c2e5-4eb9-a9c9-1d236618c9fe None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:54.334586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af72e56-c2e5-4eb9-a9c9-1d236618c9fe None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:54.334847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af72e56-c2e5-4eb9-a9c9-1d236618c9fe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:54.371913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:54.423774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af72e56-c2e5-4eb9-a9c9-1d236618c9fe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:54.425739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af72e56-c2e5-4eb9-a9c9-1d236618c9fe None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:54.563897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 616/1214] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:04:54 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 28 12:04:54.571463 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed27e35-134f-4b5a-abb5-77ecc86696fd None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:54.595549 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:54.684244 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed27e35-134f-4b5a-abb5-77ecc86696fd None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:54.719848 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e587ee-f0d1-4da7-8419-d3692f694692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:54.720506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e587ee-f0d1-4da7-8419-d3692f694692 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:54.720787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e587ee-f0d1-4da7-8419-d3692f694692 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:54.721121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e587ee-f0d1-4da7-8419-d3692f694692 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:54.732421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e587ee-f0d1-4da7-8419-d3692f694692 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:54.734682 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e587ee-f0d1-4da7-8419-d3692f694692 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-d5e587ee-f0d1-4da7-8419-d3692f694692 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:04:54.751457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:04:54.753172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:04:54.754313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:04:54.754313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:04:54.754313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:04:54.754313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 599/1215] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:04:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:04:54.758755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af72e56-c2e5-4eb9-a9c9-1d236618c9fe None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:54.830236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:54.873669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af72e56-c2e5-4eb9-a9c9-1d236618c9fe None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:54.913499 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eca224-fcc1-4b48-8c1c-e2869ad7da6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:54.914256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eca224-fcc1-4b48-8c1c-e2869ad7da6f None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:54.914584 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eca224-fcc1-4b48-8c1c-e2869ad7da6f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:54.914885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eca224-fcc1-4b48-8c1c-e2869ad7da6f None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:54.915610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29eca224-fcc1-4b48-8c1c-e2869ad7da6f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:04:54.915944 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29eca224-fcc1-4b48-8c1c-e2869ad7da6f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:54.917394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29eca224-fcc1-4b48-8c1c-e2869ad7da6f None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:54.930123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 617/1216] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:04:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:04:54.935761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e587ee-f0d1-4da7-8419-d3692f694692 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:55.022767 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:55.062981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e587ee-f0d1-4da7-8419-d3692f694692 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:55.103362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1def5245-7d13-46d3-9948-deb55d07e681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:55.104350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1def5245-7d13-46d3-9948-deb55d07e681 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:55.104678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1def5245-7d13-46d3-9948-deb55d07e681 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:55.104950 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1def5245-7d13-46d3-9948-deb55d07e681 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:55.113220 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1def5245-7d13-46d3-9948-deb55d07e681 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:55.114953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1def5245-7d13-46d3-9948-deb55d07e681 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-1def5245-7d13-46d3-9948-deb55d07e681 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:04:55.127952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:04:55.129140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:04:55.130213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:04:55.130213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:04:55.130213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:04:55.130213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:04:55.130213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:04:55.130213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:04:55.130213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:04:55.130213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:04:55.130213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 600/1217] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:04:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:04:55.136041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eca224-fcc1-4b48-8c1c-e2869ad7da6f None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:55.256582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eca224-fcc1-4b48-8c1c-e2869ad7da6f None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:55.284167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:55.297814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09434c49-c76b-41bc-9ccf-f40d15dfd2dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:55.298419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09434c49-c76b-41bc-9ccf-f40d15dfd2dd None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:55.298719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09434c49-c76b-41bc-9ccf-f40d15dfd2dd None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:55.298968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09434c49-c76b-41bc-9ccf-f40d15dfd2dd None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:55.299700 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09434c49-c76b-41bc-9ccf-f40d15dfd2dd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:04:55.300029 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09434c49-c76b-41bc-9ccf-f40d15dfd2dd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:55.301647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09434c49-c76b-41bc-9ccf-f40d15dfd2dd None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:55.315025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 618/1218] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:04:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:04:55.321194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1def5245-7d13-46d3-9948-deb55d07e681 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:55.434479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1def5245-7d13-46d3-9948-deb55d07e681 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:55.464555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:55.472143 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c64bd12-8ed9-49ec-9981-d1ee8b6bab7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:55.472896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c64bd12-8ed9-49ec-9981-d1ee8b6bab7b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:55.473150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c64bd12-8ed9-49ec-9981-d1ee8b6bab7b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:55.473555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c64bd12-8ed9-49ec-9981-d1ee8b6bab7b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:55.474348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c64bd12-8ed9-49ec-9981-d1ee8b6bab7b None admin] RBAC: Adding query filter params (role.id=ed4d262295434a72bb0a8b9a7ff12e71, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=3758ea0f26264340ad2f154c0111af21) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:04:55.474584 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c64bd12-8ed9-49ec-9981-d1ee8b6bab7b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:55.476285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c64bd12-8ed9-49ec-9981-d1ee8b6bab7b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:55.483099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 601/1219] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:04:55 2022] GET /identity/v3/role_assignments?user.id=3758ea0f26264340ad2f154c0111af21&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=ed4d262295434a72bb0a8b9a7ff12e71 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:04:57.685767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d19eae-cd15-4b51-a3d2-0215bf5d7848 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:57.686346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d19eae-cd15-4b51-a3d2-0215bf5d7848 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:57.686346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d19eae-cd15-4b51-a3d2-0215bf5d7848 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:57.688151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 619/1220] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:04:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:04:57.693511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04acddc-1136-46a0-9e57-c526136065de None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:57.693970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04acddc-1136-46a0-9e57-c526136065de None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:57.694063 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04acddc-1136-46a0-9e57-c526136065de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:57.713166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-a04acddc-1136-46a0-9e57-c526136065de None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:04:57.714620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a04acddc-1136-46a0-9e57-c526136065de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:04:57.785969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 602/1221] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:04:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:04:58.053885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d19eae-cd15-4b51-a3d2-0215bf5d7848 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:58.166832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d19eae-cd15-4b51-a3d2-0215bf5d7848 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:58.212337 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54880a40-3d8f-4204-86d7-f645ffe80b93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:58.213061 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54880a40-3d8f-4204-86d7-f645ffe80b93 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:58.213343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54880a40-3d8f-4204-86d7-f645ffe80b93 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:58.213607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54880a40-3d8f-4204-86d7-f645ffe80b93 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:58.225069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54880a40-3d8f-4204-86d7-f645ffe80b93 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-54880a40-3d8f-4204-86d7-f645ffe80b93 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:04:58.229760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:04:58.230871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:04:58.233303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:04:58.233303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:04:58.233303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:04:58.233303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:04:58.233303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:04:58.233303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:04:58.233303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:04:58.233303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:04:58.233303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:04:58.233303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:04:58.233303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:04:58.233303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:04:58.233303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 620/1222] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:04:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:04:58.236993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04acddc-1136-46a0-9e57-c526136065de None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:58.344264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04acddc-1136-46a0-9e57-c526136065de None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:58.370877 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:58.383282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c4d2bd-29e0-48f8-bb80-6689b704aa13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:58.383975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c4d2bd-29e0-48f8-bb80-6689b704aa13 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:58.385990 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c4d2bd-29e0-48f8-bb80-6689b704aa13 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:58.386277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c4d2bd-29e0-48f8-bb80-6689b704aa13 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:58.386986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c4d2bd-29e0-48f8-bb80-6689b704aa13 None admin] RBAC: Adding query filter params (name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:04:58.387334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c4d2bd-29e0-48f8-bb80-6689b704aa13 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:58.388907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c4d2bd-29e0-48f8-bb80-6689b704aa13 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:58.397119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 603/1223] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:04:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:04:58.402916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54880a40-3d8f-4204-86d7-f645ffe80b93 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:58.449774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:58.518390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54880a40-3d8f-4204-86d7-f645ffe80b93 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:58.554351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a8f2be-515b-44c5-9016-22028b91e30c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:58.555113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a8f2be-515b-44c5-9016-22028b91e30c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:58.555426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a8f2be-515b-44c5-9016-22028b91e30c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:58.555747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a8f2be-515b-44c5-9016-22028b91e30c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:58.568088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a8f2be-515b-44c5-9016-22028b91e30c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:58.570004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a8f2be-515b-44c5-9016-22028b91e30c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-26a8f2be-515b-44c5-9016-22028b91e30c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:04:58.587287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:04:58.588769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:04:58.590005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:04:58.590005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:04:58.590005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:04:58.590005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 621/1224] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:04:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:04:58.594990 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c4d2bd-29e0-48f8-bb80-6689b704aa13 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:58.711545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c4d2bd-29e0-48f8-bb80-6689b704aa13 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:58.748829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404ff6cf-ed54-4c39-a559-0c17a58a664e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:58.749455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404ff6cf-ed54-4c39-a559-0c17a58a664e None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:58.750298 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404ff6cf-ed54-4c39-a559-0c17a58a664e None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:58.750572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404ff6cf-ed54-4c39-a559-0c17a58a664e None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:58.751220 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404ff6cf-ed54-4c39-a559-0c17a58a664e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:04:58.751522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404ff6cf-ed54-4c39-a559-0c17a58a664e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:58.752927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404ff6cf-ed54-4c39-a559-0c17a58a664e None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:58.766806 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 604/1225] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:04:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:04:58.773446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a8f2be-515b-44c5-9016-22028b91e30c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:58.883589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:58.891400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a8f2be-515b-44c5-9016-22028b91e30c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:58.932871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c9effd-760c-4cd3-82e4-923fd9fe7bf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:58.933752 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c9effd-760c-4cd3-82e4-923fd9fe7bf2 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:58.934044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c9effd-760c-4cd3-82e4-923fd9fe7bf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:58.934321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c9effd-760c-4cd3-82e4-923fd9fe7bf2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:58.942881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c9effd-760c-4cd3-82e4-923fd9fe7bf2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:58.944835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c9effd-760c-4cd3-82e4-923fd9fe7bf2 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-e3c9effd-760c-4cd3-82e4-923fd9fe7bf2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:04:58.957239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:04:58.958369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:04:58.959382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:04:58.959382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:04:58.959382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:04:58.959382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:04:58.959382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:04:58.959382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:04:58.959382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:04:58.959382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:04:58.959382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 28 12:04:58.959382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:04:58.959382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 622/1226] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:04:58 2022] GET /identity/v3/users/placement => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:04:58.965119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404ff6cf-ed54-4c39-a559-0c17a58a664e None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:59.020855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:59.079370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404ff6cf-ed54-4c39-a559-0c17a58a664e None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:59.116464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78963f18-dc84-4d64-a0ca-e368ac39bf55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:59.117079 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78963f18-dc84-4d64-a0ca-e368ac39bf55 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:59.117336 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78963f18-dc84-4d64-a0ca-e368ac39bf55 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:59.117632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78963f18-dc84-4d64-a0ca-e368ac39bf55 None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:59.118374 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78963f18-dc84-4d64-a0ca-e368ac39bf55 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:04:59.118720 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78963f18-dc84-4d64-a0ca-e368ac39bf55 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:59.120222 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78963f18-dc84-4d64-a0ca-e368ac39bf55 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:59.140432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 605/1227] 10.222.0.26 () {56 vars in 1143 bytes} [Wed Sep 28 12:04:58 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:04:59.145868 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c9effd-760c-4cd3-82e4-923fd9fe7bf2 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:59.256013 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c9effd-760c-4cd3-82e4-923fd9fe7bf2 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:59.291644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732c0091-7a07-4a24-abfb-2978184e4b50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:59.292274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732c0091-7a07-4a24-abfb-2978184e4b50 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:59.292583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732c0091-7a07-4a24-abfb-2978184e4b50 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:59.293038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732c0091-7a07-4a24-abfb-2978184e4b50 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:59.304898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732c0091-7a07-4a24-abfb-2978184e4b50 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:59.306439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732c0091-7a07-4a24-abfb-2978184e4b50 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-732c0091-7a07-4a24-abfb-2978184e4b50 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:04:59.321639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:04:59.322810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:04:59.323738 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:04:59.323738 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:04:59.323738 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:04:59.323738 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 623/1228] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:04:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:04:59.329028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78963f18-dc84-4d64-a0ca-e368ac39bf55 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:59.435439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78963f18-dc84-4d64-a0ca-e368ac39bf55 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:59.438082 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:59.472787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b73c93-a2e3-4f79-807a-97d4c5b4c7fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:59.473407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b73c93-a2e3-4f79-807a-97d4c5b4c7fe None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:59.473681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b73c93-a2e3-4f79-807a-97d4c5b4c7fe None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:59.473948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b73c93-a2e3-4f79-807a-97d4c5b4c7fe None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:59.474580 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b73c93-a2e3-4f79-807a-97d4c5b4c7fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:04:59.474878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b73c93-a2e3-4f79-807a-97d4c5b4c7fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:59.476270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b73c93-a2e3-4f79-807a-97d4c5b4c7fe None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:59.494034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 606/1229] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:04:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:04:59.495262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732c0091-7a07-4a24-abfb-2978184e4b50 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:59.505507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:59.608254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732c0091-7a07-4a24-abfb-2978184e4b50 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:59.642917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbe0f17-f6cb-46c9-a961-4d785bf8451d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:59.643541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbe0f17-f6cb-46c9-a961-4d785bf8451d None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:59.643817 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbe0f17-f6cb-46c9-a961-4d785bf8451d None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:59.644088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbe0f17-f6cb-46c9-a961-4d785bf8451d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:59.650624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbe0f17-f6cb-46c9-a961-4d785bf8451d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:59.652044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbe0f17-f6cb-46c9-a961-4d785bf8451d None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-afbe0f17-f6cb-46c9-a961-4d785bf8451d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:04:59.664008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:04:59.665067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:04:59.666002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:04:59.666002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:04:59.666002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:04:59.666002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:04:59.666002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:04:59.666002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:04:59.666002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:04:59.666002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:04:59.666002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 624/1230] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:04:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:04:59.671519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b73c93-a2e3-4f79-807a-97d4c5b4c7fe None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:59.779158 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b73c93-a2e3-4f79-807a-97d4c5b4c7fe None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:59.816757 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1979a8-5a5c-4104-ab6a-c767cc4f6874 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:59.817438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1979a8-5a5c-4104-ab6a-c767cc4f6874 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:59.817728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1979a8-5a5c-4104-ab6a-c767cc4f6874 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:59.817997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1979a8-5a5c-4104-ab6a-c767cc4f6874 None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:04:59.818730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1979a8-5a5c-4104-ab6a-c767cc4f6874 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:04:59.819036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1979a8-5a5c-4104-ab6a-c767cc4f6874 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:04:59.820672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1979a8-5a5c-4104-ab6a-c767cc4f6874 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:04:59.832915 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 607/1231] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:04:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:04:59.839876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbe0f17-f6cb-46c9-a961-4d785bf8451d None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:04:59.914121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:04:59.951522 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbe0f17-f6cb-46c9-a961-4d785bf8451d None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:04:59.991572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c678a6-c346-41f1-b5b3-3ccfb2ba45b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:04:59.992317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c678a6-c346-41f1-b5b3-3ccfb2ba45b8 None admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:04:59.992720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c678a6-c346-41f1-b5b3-3ccfb2ba45b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:04:59.993129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c678a6-c346-41f1-b5b3-3ccfb2ba45b8 None admin] PATH_INFO: `/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/3758ea0f26264340ad2f154c0111af21/roles/ed4d262295434a72bb0a8b9a7ff12e71` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:00.008646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c678a6-c346-41f1-b5b3-3ccfb2ba45b8 None admin] RBAC: Authorizing `identity:create_grant(project_id=8695d562e3e24c7c9300cf3a23cd7502, user_id=3758ea0f26264340ad2f154c0111af21, role_id=ed4d262295434a72bb0a8b9a7ff12e71)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:00.010449 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c678a6-c346-41f1-b5b3-3ccfb2ba45b8 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:00.023422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 625/1232] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:04:59 2022] PUT /identity/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/3758ea0f26264340ad2f154c0111af21/roles/ed4d262295434a72bb0a8b9a7ff12e71 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:05:02.249286 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d145fb8-e3fa-4b2a-b70c-9866b7204676 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:02.250038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d145fb8-e3fa-4b2a-b70c-9866b7204676 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:02.250038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d145fb8-e3fa-4b2a-b70c-9866b7204676 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:02.251196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 608/1233] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:05:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:05:02.256671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3473988-624d-4b9f-91dc-15dedc5ccfce None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:02.257143 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3473988-624d-4b9f-91dc-15dedc5ccfce None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:02.257254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3473988-624d-4b9f-91dc-15dedc5ccfce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:02.276068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-b3473988-624d-4b9f-91dc-15dedc5ccfce None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:05:02.277360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b3473988-624d-4b9f-91dc-15dedc5ccfce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:05:02.351482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 626/1234] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:05:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:05:02.631686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d145fb8-e3fa-4b2a-b70c-9866b7204676 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:02.660463 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:02.756469 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d145fb8-e3fa-4b2a-b70c-9866b7204676 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:02.793805 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7067ad24-ae8e-494c-8366-019a51551edc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:02.794627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7067ad24-ae8e-494c-8366-019a51551edc None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:02.794956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7067ad24-ae8e-494c-8366-019a51551edc None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:02.795575 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7067ad24-ae8e-494c-8366-019a51551edc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:02.808321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7067ad24-ae8e-494c-8366-019a51551edc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-7067ad24-ae8e-494c-8366-019a51551edc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:05:02.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:02.815754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:02.816880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:02.816880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:02.816880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:02.816880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:02.816880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:05:02.816880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:05:02.816880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:05:02.816880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:05:02.816880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:05:02.816880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:05:02.816880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 28 12:05:02.816880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:05:02.816880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 609/1235] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Sep 28 12:05:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 28 12:05:02.822257 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3473988-624d-4b9f-91dc-15dedc5ccfce None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:02.871663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:02.932397 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3473988-624d-4b9f-91dc-15dedc5ccfce None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:02.971516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9de17e-a014-4863-939d-c6a02bbb4c0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:02.972175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9de17e-a014-4863-939d-c6a02bbb4c0e None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:02.972583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9de17e-a014-4863-939d-c6a02bbb4c0e None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:02.972943 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9de17e-a014-4863-939d-c6a02bbb4c0e None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:02.973765 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9de17e-a014-4863-939d-c6a02bbb4c0e None admin] RBAC: Adding query filter params (name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:02.974052 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9de17e-a014-4863-939d-c6a02bbb4c0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:02.975693 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9de17e-a014-4863-939d-c6a02bbb4c0e None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:02.985577 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 627/1236] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Sep 28 12:05:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:02.992105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7067ad24-ae8e-494c-8366-019a51551edc None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:03.087752 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:03.105725 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7067ad24-ae8e-494c-8366-019a51551edc None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:03.142152 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88b1239-76b0-441c-a50e-9dc0f37e69ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:03.142784 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88b1239-76b0-441c-a50e-9dc0f37e69ad None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:03.143062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88b1239-76b0-441c-a50e-9dc0f37e69ad None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:03.143329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88b1239-76b0-441c-a50e-9dc0f37e69ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:03.154105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88b1239-76b0-441c-a50e-9dc0f37e69ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:03.155548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88b1239-76b0-441c-a50e-9dc0f37e69ad None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-b88b1239-76b0-441c-a50e-9dc0f37e69ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:05:03.171278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:03.172277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:05:03.173433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:05:03.173433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:03.173433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:05:03.173433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 610/1237] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:05:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:03.178881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9de17e-a014-4863-939d-c6a02bbb4c0e None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:03.292133 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9de17e-a014-4863-939d-c6a02bbb4c0e None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:03.301687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:03.326582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbeaa7dc-3577-437d-b928-a21cea73dd52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:03.327187 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbeaa7dc-3577-437d-b928-a21cea73dd52 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:03.327481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbeaa7dc-3577-437d-b928-a21cea73dd52 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:03.327760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbeaa7dc-3577-437d-b928-a21cea73dd52 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:03.328438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbeaa7dc-3577-437d-b928-a21cea73dd52 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:03.328745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbeaa7dc-3577-437d-b928-a21cea73dd52 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:03.330111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbeaa7dc-3577-437d-b928-a21cea73dd52 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:03.341881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 628/1238] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:05:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:03.348245 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88b1239-76b0-441c-a50e-9dc0f37e69ad None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:03.460711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88b1239-76b0-441c-a50e-9dc0f37e69ad None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:03.497597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfdfecd-53ea-4fec-9cdf-d03151088f75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:03.498347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfdfecd-53ea-4fec-9cdf-d03151088f75 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:03.500160 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfdfecd-53ea-4fec-9cdf-d03151088f75 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:03.500473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfdfecd-53ea-4fec-9cdf-d03151088f75 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:03.506381 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:03.512004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dfdfecd-53ea-4fec-9cdf-d03151088f75 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:03.513705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dfdfecd-53ea-4fec-9cdf-d03151088f75 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-6dfdfecd-53ea-4fec-9cdf-d03151088f75 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:05:03.527190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:03.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:03.534678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:03.534678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:03.534678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:05:03.534678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:05:03.534678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:05:03.534678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:05:03.534678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:05:03.534678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:05:03.534678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 28 12:05:03.534678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:05:03.534678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 611/1239] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:05:03 2022] GET /identity/v3/users/placement => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:03.535272 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbeaa7dc-3577-437d-b928-a21cea73dd52 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:03.641123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbeaa7dc-3577-437d-b928-a21cea73dd52 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:03.678809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea20e234-8fef-4ac0-bd3e-870e6657fe8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:03.679408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea20e234-8fef-4ac0-bd3e-870e6657fe8c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:03.679683 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea20e234-8fef-4ac0-bd3e-870e6657fe8c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:03.679952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea20e234-8fef-4ac0-bd3e-870e6657fe8c None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:03.680713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea20e234-8fef-4ac0-bd3e-870e6657fe8c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:03.681017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea20e234-8fef-4ac0-bd3e-870e6657fe8c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:03.682468 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea20e234-8fef-4ac0-bd3e-870e6657fe8c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:03.700446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 629/1240] 10.222.0.26 () {56 vars in 1143 bytes} [Wed Sep 28 12:05:03 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:03.706573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfdfecd-53ea-4fec-9cdf-d03151088f75 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:03.817341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfdfecd-53ea-4fec-9cdf-d03151088f75 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:03.854072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d3bbef-237b-4a5d-8485-8878a2debc23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:03.854796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d3bbef-237b-4a5d-8485-8878a2debc23 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:03.855121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d3bbef-237b-4a5d-8485-8878a2debc23 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:03.855435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d3bbef-237b-4a5d-8485-8878a2debc23 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:03.923197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:03.931128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d3bbef-237b-4a5d-8485-8878a2debc23 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:03.932684 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d3bbef-237b-4a5d-8485-8878a2debc23 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:04.045680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 612/1241] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:05:03 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:05:04.051630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea20e234-8fef-4ac0-bd3e-870e6657fe8c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:04.080850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:04.163373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea20e234-8fef-4ac0-bd3e-870e6657fe8c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:04.202138 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e032a6c6-4a41-4bf0-9d29-362d5bb22e8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:04.203201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e032a6c6-4a41-4bf0-9d29-362d5bb22e8d None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:04.203532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e032a6c6-4a41-4bf0-9d29-362d5bb22e8d None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:04.203835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e032a6c6-4a41-4bf0-9d29-362d5bb22e8d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:04.215564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e032a6c6-4a41-4bf0-9d29-362d5bb22e8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:04.217133 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e032a6c6-4a41-4bf0-9d29-362d5bb22e8d None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-e032a6c6-4a41-4bf0-9d29-362d5bb22e8d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:05:04.234075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:04.235233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:05:04.236140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:05:04.236140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:04.236140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:04.236140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 630/1242] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:05:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:04.241469 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d3bbef-237b-4a5d-8485-8878a2debc23 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:04.348541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:04.355278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d3bbef-237b-4a5d-8485-8878a2debc23 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:04.394149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb89c5c8-b742-49e9-9939-1de0040bd661 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:04.394772 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb89c5c8-b742-49e9-9939-1de0040bd661 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:04.395078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb89c5c8-b742-49e9-9939-1de0040bd661 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:04.395352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb89c5c8-b742-49e9-9939-1de0040bd661 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:04.396003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb89c5c8-b742-49e9-9939-1de0040bd661 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:04.396322 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb89c5c8-b742-49e9-9939-1de0040bd661 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:04.397718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb89c5c8-b742-49e9-9939-1de0040bd661 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:04.412412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 613/1243] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:05:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:04.418628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e032a6c6-4a41-4bf0-9d29-362d5bb22e8d None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:04.506317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:04.529881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e032a6c6-4a41-4bf0-9d29-362d5bb22e8d None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:04.566230 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7320181a-261c-4cf4-b265-bdde44f25772 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:04.566856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7320181a-261c-4cf4-b265-bdde44f25772 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:04.567132 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7320181a-261c-4cf4-b265-bdde44f25772 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:04.567411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7320181a-261c-4cf4-b265-bdde44f25772 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:04.573991 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7320181a-261c-4cf4-b265-bdde44f25772 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:04.575430 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7320181a-261c-4cf4-b265-bdde44f25772 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-7320181a-261c-4cf4-b265-bdde44f25772 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:05:04.588820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:04.590328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:04.591726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:05:04.591726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:05:04.591726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:05:04.591726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:05:04.591726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:05:04.591726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:05:04.591726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:05:04.591726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:04.591726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 631/1244] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:05:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:04.596398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb89c5c8-b742-49e9-9939-1de0040bd661 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:04.708498 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb89c5c8-b742-49e9-9939-1de0040bd661 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:04.746138 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a16a0bd-e090-45e1-813a-8c5475095be9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:04.746661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a16a0bd-e090-45e1-813a-8c5475095be9 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:04.746942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a16a0bd-e090-45e1-813a-8c5475095be9 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:04.747224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a16a0bd-e090-45e1-813a-8c5475095be9 None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:04.747905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a16a0bd-e090-45e1-813a-8c5475095be9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:04.748230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a16a0bd-e090-45e1-813a-8c5475095be9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:04.749817 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a16a0bd-e090-45e1-813a-8c5475095be9 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:04.761731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 614/1245] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:05:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:04.768402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7320181a-261c-4cf4-b265-bdde44f25772 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:04.873618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7320181a-261c-4cf4-b265-bdde44f25772 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:04.911084 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee7ed11-33d5-46ed-a340-1b5d1f32809c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:04.911745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee7ed11-33d5-46ed-a340-1b5d1f32809c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:04.912143 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee7ed11-33d5-46ed-a340-1b5d1f32809c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:04.912413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee7ed11-33d5-46ed-a340-1b5d1f32809c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:04.913232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee7ed11-33d5-46ed-a340-1b5d1f32809c None admin] RBAC: Adding query filter params (role.id=ed4d262295434a72bb0a8b9a7ff12e71, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=3758ea0f26264340ad2f154c0111af21) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:04.913624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee7ed11-33d5-46ed-a340-1b5d1f32809c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:04.915194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee7ed11-33d5-46ed-a340-1b5d1f32809c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:04.922680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 632/1246] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:05:04 2022] GET /identity/v3/role_assignments?user.id=3758ea0f26264340ad2f154c0111af21&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=ed4d262295434a72bb0a8b9a7ff12e71 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:07.261521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c9bc68-c8a9-4942-8ca2-ab3baf5a9f9e None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:07.262094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c9bc68-c8a9-4942-8ca2-ab3baf5a9f9e None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:07.262302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c9bc68-c8a9-4942-8ca2-ab3baf5a9f9e None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:07.264340 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 615/1247] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:05:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:05:07.269800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b32921-1773-4fe9-b2a9-a7daa2b60e64 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:07.270302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b32921-1773-4fe9-b2a9-a7daa2b60e64 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:07.270390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b32921-1773-4fe9-b2a9-a7daa2b60e64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:07.276675 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:07.289340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-64b32921-1773-4fe9-b2a9-a7daa2b60e64 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:05:07.290649 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-64b32921-1773-4fe9-b2a9-a7daa2b60e64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:05:07.361424 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 633/1248] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:05:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 28 12:05:07.654781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c9bc68-c8a9-4942-8ca2-ab3baf5a9f9e None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:07.673621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:07.761744 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c9bc68-c8a9-4942-8ca2-ab3baf5a9f9e None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:07.798512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99567918-163c-4b95-90ca-6adb66f6de85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:07.799238 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99567918-163c-4b95-90ca-6adb66f6de85 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:07.799603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99567918-163c-4b95-90ca-6adb66f6de85 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:07.799861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99567918-163c-4b95-90ca-6adb66f6de85 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:07.810634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99567918-163c-4b95-90ca-6adb66f6de85 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-99567918-163c-4b95-90ca-6adb66f6de85 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:05:07.815967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:07.817126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:07.818051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:07.818051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:07.818051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:07.818051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:07.818051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:05:07.818051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:05:07.818051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:05:07.818051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:05:07.818051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:05:07.818051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:05:07.818051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:05:07.818051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:05:07.818051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 616/1249] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:05:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:07.823062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b32921-1773-4fe9-b2a9-a7daa2b60e64 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:07.926717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b32921-1773-4fe9-b2a9-a7daa2b60e64 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:07.957158 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:07.962846 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8da696-6c03-4ae4-a160-5abd35ea899c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:07.963496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8da696-6c03-4ae4-a160-5abd35ea899c None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:07.963776 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8da696-6c03-4ae4-a160-5abd35ea899c None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:07.964048 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8da696-6c03-4ae4-a160-5abd35ea899c None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:07.965003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8da696-6c03-4ae4-a160-5abd35ea899c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:07.965325 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8da696-6c03-4ae4-a160-5abd35ea899c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:07.966663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8da696-6c03-4ae4-a160-5abd35ea899c None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:07.974898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 634/1250] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:05:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:07.981224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99567918-163c-4b95-90ca-6adb66f6de85 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:08.064751 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:08.089728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99567918-163c-4b95-90ca-6adb66f6de85 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:08.127173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e21c3b-7332-4849-9fa6-248ca7a49e91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:08.128057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e21c3b-7332-4849-9fa6-248ca7a49e91 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:08.128607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e21c3b-7332-4849-9fa6-248ca7a49e91 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:08.128930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e21c3b-7332-4849-9fa6-248ca7a49e91 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:08.139704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e21c3b-7332-4849-9fa6-248ca7a49e91 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:08.141444 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e21c3b-7332-4849-9fa6-248ca7a49e91 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-b8e21c3b-7332-4849-9fa6-248ca7a49e91 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:05:08.159189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:08.160337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:05:08.161287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:05:08.161287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:08.161287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:05:08.161287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 617/1251] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:05:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:08.166395 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8da696-6c03-4ae4-a160-5abd35ea899c None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:08.278832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8da696-6c03-4ae4-a160-5abd35ea899c None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:08.315483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cc02e5-f123-4bb0-a179-3b5f0d314896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:08.316107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cc02e5-f123-4bb0-a179-3b5f0d314896 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:08.316483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cc02e5-f123-4bb0-a179-3b5f0d314896 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:08.316798 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cc02e5-f123-4bb0-a179-3b5f0d314896 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:08.317481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cc02e5-f123-4bb0-a179-3b5f0d314896 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:08.317799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cc02e5-f123-4bb0-a179-3b5f0d314896 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:08.319280 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cc02e5-f123-4bb0-a179-3b5f0d314896 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:08.330896 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 635/1252] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:05:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:05:08.337150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e21c3b-7332-4849-9fa6-248ca7a49e91 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:08.451043 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e21c3b-7332-4849-9fa6-248ca7a49e91 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:08.494618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:08.503630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c715abd8-f71c-49b7-bfc4-7a00990bc38b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:08.504281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c715abd8-f71c-49b7-bfc4-7a00990bc38b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:08.504593 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c715abd8-f71c-49b7-bfc4-7a00990bc38b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:08.504872 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c715abd8-f71c-49b7-bfc4-7a00990bc38b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:08.513660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c715abd8-f71c-49b7-bfc4-7a00990bc38b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:08.515708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c715abd8-f71c-49b7-bfc4-7a00990bc38b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-c715abd8-f71c-49b7-bfc4-7a00990bc38b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:05:08.528053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:08.529373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:08.531403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:08.531403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:08.531403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:05:08.531403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:05:08.531403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:05:08.531403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:05:08.531403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:05:08.531403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:05:08.531403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 28 12:05:08.531403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:05:08.531403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 618/1253] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:05:08 2022] GET /identity/v3/users/placement => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:08.535873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cc02e5-f123-4bb0-a179-3b5f0d314896 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:08.593898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:08.639409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cc02e5-f123-4bb0-a179-3b5f0d314896 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:08.676606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf34a2d-91fc-41cd-84f0-b8ed9e101ca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:08.677213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf34a2d-91fc-41cd-84f0-b8ed9e101ca4 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:08.677493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf34a2d-91fc-41cd-84f0-b8ed9e101ca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:08.677766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf34a2d-91fc-41cd-84f0-b8ed9e101ca4 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:08.678503 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf34a2d-91fc-41cd-84f0-b8ed9e101ca4 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:08.678808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf34a2d-91fc-41cd-84f0-b8ed9e101ca4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:08.680240 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf34a2d-91fc-41cd-84f0-b8ed9e101ca4 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:08.699398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 636/1254] 10.222.0.26 () {56 vars in 1143 bytes} [Wed Sep 28 12:05:08 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:08.706366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c715abd8-f71c-49b7-bfc4-7a00990bc38b None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:08.814953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c715abd8-f71c-49b7-bfc4-7a00990bc38b None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:08.850315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49555591-47ba-41ac-a201-ffb91c3bba75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:08.850925 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49555591-47ba-41ac-a201-ffb91c3bba75 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:08.851202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49555591-47ba-41ac-a201-ffb91c3bba75 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:08.851473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49555591-47ba-41ac-a201-ffb91c3bba75 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:08.908542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:08.920909 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49555591-47ba-41ac-a201-ffb91c3bba75 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:08.922430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49555591-47ba-41ac-a201-ffb91c3bba75 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:09.020010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 619/1255] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:05:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:05:09.026063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf34a2d-91fc-41cd-84f0-b8ed9e101ca4 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:09.133676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf34a2d-91fc-41cd-84f0-b8ed9e101ca4 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:09.148487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:09.172555 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e62fad-22d3-4cd0-9fa6-bb00ecfb7009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:09.173257 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e62fad-22d3-4cd0-9fa6-bb00ecfb7009 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:09.173561 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e62fad-22d3-4cd0-9fa6-bb00ecfb7009 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:09.173885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e62fad-22d3-4cd0-9fa6-bb00ecfb7009 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:09.185238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e62fad-22d3-4cd0-9fa6-bb00ecfb7009 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:09.186717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e62fad-22d3-4cd0-9fa6-bb00ecfb7009 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-67e62fad-22d3-4cd0-9fa6-bb00ecfb7009 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:05:09.200699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:09.201791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:05:09.203528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:05:09.203528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:09.203528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:09.203528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 637/1256] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:05:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:09.208080 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49555591-47ba-41ac-a201-ffb91c3bba75 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:09.292756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:09.304765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49555591-47ba-41ac-a201-ffb91c3bba75 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:09.338179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3234324-8c43-4f58-9128-5aeabf228a2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:09.338770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3234324-8c43-4f58-9128-5aeabf228a2c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:09.339041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3234324-8c43-4f58-9128-5aeabf228a2c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:09.339313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3234324-8c43-4f58-9128-5aeabf228a2c None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:09.340203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3234324-8c43-4f58-9128-5aeabf228a2c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:09.340537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3234324-8c43-4f58-9128-5aeabf228a2c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:09.341898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3234324-8c43-4f58-9128-5aeabf228a2c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:09.353227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 620/1257] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:05:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:09.358860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e62fad-22d3-4cd0-9fa6-bb00ecfb7009 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:09.459611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e62fad-22d3-4cd0-9fa6-bb00ecfb7009 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:09.492215 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08323e19-97a3-4da5-80ef-99767c252579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:09.493338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08323e19-97a3-4da5-80ef-99767c252579 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:09.493338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08323e19-97a3-4da5-80ef-99767c252579 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:09.493493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08323e19-97a3-4da5-80ef-99767c252579 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:09.500372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08323e19-97a3-4da5-80ef-99767c252579 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:09.501810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08323e19-97a3-4da5-80ef-99767c252579 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-08323e19-97a3-4da5-80ef-99767c252579 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:05:09.512668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:09.513671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:09.514746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:05:09.514746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:05:09.514746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:05:09.514746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:05:09.514746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:05:09.514746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:05:09.514746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:05:09.514746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:09.514746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 638/1258] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:05:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:09.520155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3234324-8c43-4f58-9128-5aeabf228a2c None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:09.615336 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3234324-8c43-4f58-9128-5aeabf228a2c None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:09.648353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec1d0de-7137-42d9-b93f-c94475cf370b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:09.648948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec1d0de-7137-42d9-b93f-c94475cf370b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:09.649230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec1d0de-7137-42d9-b93f-c94475cf370b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:09.649507 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec1d0de-7137-42d9-b93f-c94475cf370b None admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:09.650190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec1d0de-7137-42d9-b93f-c94475cf370b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:09.650495 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec1d0de-7137-42d9-b93f-c94475cf370b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:09.651911 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec1d0de-7137-42d9-b93f-c94475cf370b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:09.663422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 621/1259] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:05:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:09.669343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08323e19-97a3-4da5-80ef-99767c252579 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:09.694263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:09.770498 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08323e19-97a3-4da5-80ef-99767c252579 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:09.802499 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e52986c-408b-4ab5-9605-ba16c02ceadb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:09.803094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e52986c-408b-4ab5-9605-ba16c02ceadb None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:09.803373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e52986c-408b-4ab5-9605-ba16c02ceadb None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:09.803646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e52986c-408b-4ab5-9605-ba16c02ceadb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:09.804400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e52986c-408b-4ab5-9605-ba16c02ceadb None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=3758ea0f26264340ad2f154c0111af21) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:09.804712 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e52986c-408b-4ab5-9605-ba16c02ceadb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:09.806116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e52986c-408b-4ab5-9605-ba16c02ceadb None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:09.811248 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 639/1260] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:05:09 2022] GET /identity/v3/role_assignments?user.id=3758ea0f26264340ad2f154c0111af21&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 259 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:11.929431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a7397b-942f-44f6-b09b-651c3b939a60 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:11.929994 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a7397b-942f-44f6-b09b-651c3b939a60 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:11.929994 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a7397b-942f-44f6-b09b-651c3b939a60 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:11.931291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 622/1261] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:05:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:05:11.936253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d074b5f-aa87-4b4e-86c8-23c822fad110 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:11.937195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d074b5f-aa87-4b4e-86c8-23c822fad110 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:11.937302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d074b5f-aa87-4b4e-86c8-23c822fad110 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:11.955356 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4d074b5f-aa87-4b4e-86c8-23c822fad110 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:05:11.956552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4d074b5f-aa87-4b4e-86c8-23c822fad110 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:05:12.019906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 640/1262] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:05:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:05:12.287776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a7397b-942f-44f6-b09b-651c3b939a60 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:12.298550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:12.390836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a7397b-942f-44f6-b09b-651c3b939a60 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:12.427622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e70969-7a27-4c40-bfdc-dbea29d75d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:12.428236 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e70969-7a27-4c40-bfdc-dbea29d75d69 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:12.428539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e70969-7a27-4c40-bfdc-dbea29d75d69 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:12.428804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e70969-7a27-4c40-bfdc-dbea29d75d69 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:12.438847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e70969-7a27-4c40-bfdc-dbea29d75d69 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-e7e70969-7a27-4c40-bfdc-dbea29d75d69 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:05:12.443564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:12.444650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:12.445747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:12.445747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:12.445747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:12.445747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:12.445747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:05:12.445747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:05:12.445747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:05:12.445747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:05:12.445747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:05:12.445747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:05:12.445747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:05:12.445747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:05:12.445747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 623/1263] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:05:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:12.453179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d074b5f-aa87-4b4e-86c8-23c822fad110 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:12.471490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:12.550403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d074b5f-aa87-4b4e-86c8-23c822fad110 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:12.583728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13621ab-1919-462b-805f-f5e1b4dd8cc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:12.584343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13621ab-1919-462b-805f-f5e1b4dd8cc1 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:12.584630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13621ab-1919-462b-805f-f5e1b4dd8cc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:12.584907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13621ab-1919-462b-805f-f5e1b4dd8cc1 None admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:12.585563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13621ab-1919-462b-805f-f5e1b4dd8cc1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:12.585867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13621ab-1919-462b-805f-f5e1b4dd8cc1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:12.587221 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13621ab-1919-462b-805f-f5e1b4dd8cc1 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:12.595275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 641/1264] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:05:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:12.600961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e70969-7a27-4c40-bfdc-dbea29d75d69 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:12.665523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:12.706816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e70969-7a27-4c40-bfdc-dbea29d75d69 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:12.743986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a812d2-46a1-49a9-ad30-c0b4109cfefb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:12.744634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a812d2-46a1-49a9-ad30-c0b4109cfefb None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:12.744915 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a812d2-46a1-49a9-ad30-c0b4109cfefb None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:12.745186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a812d2-46a1-49a9-ad30-c0b4109cfefb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:12.755373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a812d2-46a1-49a9-ad30-c0b4109cfefb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:12.756845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a812d2-46a1-49a9-ad30-c0b4109cfefb None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-c9a812d2-46a1-49a9-ad30-c0b4109cfefb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:05:12.771405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:12.772551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:05:12.773594 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:05:12.773594 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:12.773594 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:05:12.773594 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 624/1265] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:05:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:12.778268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13621ab-1919-462b-805f-f5e1b4dd8cc1 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:12.855123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:12.874775 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13621ab-1919-462b-805f-f5e1b4dd8cc1 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:12.906067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aaba97-78a7-473f-b8ea-774a5b202393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:12.906655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aaba97-78a7-473f-b8ea-774a5b202393 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:12.906933 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aaba97-78a7-473f-b8ea-774a5b202393 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:12.907205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aaba97-78a7-473f-b8ea-774a5b202393 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:12.907836 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74aaba97-78a7-473f-b8ea-774a5b202393 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:12.908138 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74aaba97-78a7-473f-b8ea-774a5b202393 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:12.909499 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74aaba97-78a7-473f-b8ea-774a5b202393 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:12.920606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 642/1266] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:05:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:12.926621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a812d2-46a1-49a9-ad30-c0b4109cfefb None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:13.027315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a812d2-46a1-49a9-ad30-c0b4109cfefb None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:13.046636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:13.060771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbff7d97-71e9-4511-8950-a7f48a36acfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:13.061393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbff7d97-71e9-4511-8950-a7f48a36acfe None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:13.062252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbff7d97-71e9-4511-8950-a7f48a36acfe None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:13.062537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbff7d97-71e9-4511-8950-a7f48a36acfe None admin] PATH_INFO: `/v3/users/placement` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:13.070573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbff7d97-71e9-4511-8950-a7f48a36acfe None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:13.072011 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbff7d97-71e9-4511-8950-a7f48a36acfe None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-fbff7d97-71e9-4511-8950-a7f48a36acfe None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:05:13.082763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:13.083854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:13.084846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:13.084846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:13.084846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:05:13.084846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:05:13.084846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:05:13.084846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:05:13.084846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:05:13.084846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:05:13.084846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 28 12:05:13.084846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:05:13.084846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 625/1267] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:05:12 2022] GET /identity/v3/users/placement => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:13.089972 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aaba97-78a7-473f-b8ea-774a5b202393 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:13.192033 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aaba97-78a7-473f-b8ea-774a5b202393 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:13.224232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4af107-7133-41ff-8ffa-5df3c107c759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:13.224855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4af107-7133-41ff-8ffa-5df3c107c759 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:13.225136 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4af107-7133-41ff-8ffa-5df3c107c759 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:13.225409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4af107-7133-41ff-8ffa-5df3c107c759 None admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:13.226132 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4af107-7133-41ff-8ffa-5df3c107c759 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:13.226436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4af107-7133-41ff-8ffa-5df3c107c759 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:13.227838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4af107-7133-41ff-8ffa-5df3c107c759 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:13.236559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:13.245064 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 643/1268] 10.222.0.26 () {56 vars in 1143 bytes} [Wed Sep 28 12:05:13 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:13.251556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbff7d97-71e9-4511-8950-a7f48a36acfe None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:13.347575 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbff7d97-71e9-4511-8950-a7f48a36acfe None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:13.379399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0e17b0-3d23-4ce4-b873-73ebf4515e0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:13.380001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0e17b0-3d23-4ce4-b873-73ebf4515e0c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:13.380277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0e17b0-3d23-4ce4-b873-73ebf4515e0c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:13.380636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0e17b0-3d23-4ce4-b873-73ebf4515e0c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:13.391161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0e17b0-3d23-4ce4-b873-73ebf4515e0c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:13.392619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0e17b0-3d23-4ce4-b873-73ebf4515e0c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-ec0e17b0-3d23-4ce4-b873-73ebf4515e0c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:05:13.406128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:13.407246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:05:13.408562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:05:13.408562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:13.408562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:05:13.408562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 626/1269] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:05:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:13.412851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4af107-7133-41ff-8ffa-5df3c107c759 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:13.506789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4af107-7133-41ff-8ffa-5df3c107c759 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:13.539464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71afd9d9-9372-4743-b947-32b2711236d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:13.540054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71afd9d9-9372-4743-b947-32b2711236d0 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:13.540354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71afd9d9-9372-4743-b947-32b2711236d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:13.540628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71afd9d9-9372-4743-b947-32b2711236d0 None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:13.541263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71afd9d9-9372-4743-b947-32b2711236d0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:13.541563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71afd9d9-9372-4743-b947-32b2711236d0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:13.542911 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71afd9d9-9372-4743-b947-32b2711236d0 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:13.553536 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 644/1270] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:05:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:05:13.559262 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0e17b0-3d23-4ce4-b873-73ebf4515e0c None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:13.590620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:13.656128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0e17b0-3d23-4ce4-b873-73ebf4515e0c None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:13.687299 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2957991e-762d-42b4-8a30-2848c77f0c0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:13.687899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2957991e-762d-42b4-8a30-2848c77f0c0f None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:13.688174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2957991e-762d-42b4-8a30-2848c77f0c0f None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:13.688472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2957991e-762d-42b4-8a30-2848c77f0c0f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:13.694726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2957991e-762d-42b4-8a30-2848c77f0c0f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:13.696149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2957991e-762d-42b4-8a30-2848c77f0c0f None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-2957991e-762d-42b4-8a30-2848c77f0c0f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:05:13.707455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:13.708552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:13.709570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:05:13.709570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:05:13.709570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:05:13.709570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:05:13.709570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:05:13.709570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:05:13.709570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:05:13.709570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:05:13.709570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 627/1271] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:05:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:13.715001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71afd9d9-9372-4743-b947-32b2711236d0 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:13.773331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:13.816532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71afd9d9-9372-4743-b947-32b2711236d0 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:13.850726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d241ac0-0171-4296-a1dd-4391b3a054ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:13.851330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d241ac0-0171-4296-a1dd-4391b3a054ca None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:13.851609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d241ac0-0171-4296-a1dd-4391b3a054ca None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:13.851881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d241ac0-0171-4296-a1dd-4391b3a054ca None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:13.852607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d241ac0-0171-4296-a1dd-4391b3a054ca None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:13.852915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d241ac0-0171-4296-a1dd-4391b3a054ca None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:13.854340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d241ac0-0171-4296-a1dd-4391b3a054ca None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:13.866058 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 645/1272] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:05:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:13.872341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2957991e-762d-42b4-8a30-2848c77f0c0f None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:13.961669 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:13.971406 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2957991e-762d-42b4-8a30-2848c77f0c0f None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:14.004746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa0399a-3c62-4cd5-9265-37a17d0b964c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:14.005396 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa0399a-3c62-4cd5-9265-37a17d0b964c None admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:14.005672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa0399a-3c62-4cd5-9265-37a17d0b964c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:14.005941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa0399a-3c62-4cd5-9265-37a17d0b964c None admin] PATH_INFO: `/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/3758ea0f26264340ad2f154c0111af21/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:14.007810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa0399a-3c62-4cd5-9265-37a17d0b964c None admin] RBAC: Authorizing `identity:create_grant(project_id=8695d562e3e24c7c9300cf3a23cd7502, user_id=3758ea0f26264340ad2f154c0111af21, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:14.009267 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa0399a-3c62-4cd5-9265-37a17d0b964c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:14.021433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 628/1273] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Sep 28 12:05:13 2022] PUT /identity/v3/projects/8695d562e3e24c7c9300cf3a23cd7502/users/3758ea0f26264340ad2f154c0111af21/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:05:16.116861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c986613-6812-4daf-86f8-da97651399b4 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:16.117393 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c986613-6812-4daf-86f8-da97651399b4 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:16.117393 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c986613-6812-4daf-86f8-da97651399b4 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:16.118717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 646/1274] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:05:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:05:16.123979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1ccf94-f0e8-4cb0-9c8b-84933002b0a2 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:16.124438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1ccf94-f0e8-4cb0-9c8b-84933002b0a2 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:16.124668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1ccf94-f0e8-4cb0-9c8b-84933002b0a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:16.143058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-ff1ccf94-f0e8-4cb0-9c8b-84933002b0a2 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:05:16.144263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-ff1ccf94-f0e8-4cb0-9c8b-84933002b0a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:05:16.216569 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 629/1275] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:05:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:05:16.487653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c986613-6812-4daf-86f8-da97651399b4 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:16.584714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c986613-6812-4daf-86f8-da97651399b4 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:16.601901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:16.619159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9019d0-3d79-4bb1-968d-2a139e1c860d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:16.619775 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9019d0-3d79-4bb1-968d-2a139e1c860d None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:16.620054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9019d0-3d79-4bb1-968d-2a139e1c860d None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:16.620412 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9019d0-3d79-4bb1-968d-2a139e1c860d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:16.630125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9019d0-3d79-4bb1-968d-2a139e1c860d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-ee9019d0-3d79-4bb1-968d-2a139e1c860d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 28 12:05:16.634969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:16.636076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:16.637062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:16.637062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:16.637062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:16.637062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:16.637062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 28 12:05:16.637062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 28 12:05:16.637062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 28 12:05:16.637062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 28 12:05:16.637062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 28 12:05:16.637062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 28 12:05:16.637062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 28 12:05:16.637062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:16.637062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 647/1276] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Sep 28 12:05:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:16.642597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1ccf94-f0e8-4cb0-9c8b-84933002b0a2 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:16.720323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:16.741880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1ccf94-f0e8-4cb0-9c8b-84933002b0a2 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:16.774892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d69253-07ae-48ef-9cec-fa876e48d827 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:16.775479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d69253-07ae-48ef-9cec-fa876e48d827 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:16.775753 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d69253-07ae-48ef-9cec-fa876e48d827 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:16.776019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d69253-07ae-48ef-9cec-fa876e48d827 None admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:16.776700 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d69253-07ae-48ef-9cec-fa876e48d827 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:16.776997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d69253-07ae-48ef-9cec-fa876e48d827 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:16.778349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d69253-07ae-48ef-9cec-fa876e48d827 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:16.786509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 630/1277] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Sep 28 12:05:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:16.792323 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9019d0-3d79-4bb1-968d-2a139e1c860d None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:16.886294 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9019d0-3d79-4bb1-968d-2a139e1c860d None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:16.918719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6207c1e2-dbda-4fa4-b6a4-195e27245a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:16.919319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6207c1e2-dbda-4fa4-b6a4-195e27245a69 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:16.919591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6207c1e2-dbda-4fa4-b6a4-195e27245a69 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:16.919857 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6207c1e2-dbda-4fa4-b6a4-195e27245a69 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:16.929300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6207c1e2-dbda-4fa4-b6a4-195e27245a69 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:16.930718 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6207c1e2-dbda-4fa4-b6a4-195e27245a69 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-6207c1e2-dbda-4fa4-b6a4-195e27245a69 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:05:16.944451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:16.945623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:05:16.946729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:05:16.946729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:16.946729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:16.946729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 648/1278] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:05:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:16.951025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d69253-07ae-48ef-9cec-fa876e48d827 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:17.047845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d69253-07ae-48ef-9cec-fa876e48d827 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:17.079434 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d76b7d-07fa-489c-938e-e1e3cc742122 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:17.080019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d76b7d-07fa-489c-938e-e1e3cc742122 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:17.080297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d76b7d-07fa-489c-938e-e1e3cc742122 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:17.080596 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d76b7d-07fa-489c-938e-e1e3cc742122 None admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:17.081228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d76b7d-07fa-489c-938e-e1e3cc742122 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:17.081528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d76b7d-07fa-489c-938e-e1e3cc742122 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:17.082868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d76b7d-07fa-489c-938e-e1e3cc742122 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:17.094818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 631/1279] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:05:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:17.100283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6207c1e2-dbda-4fa4-b6a4-195e27245a69 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:17.129765 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:17.200872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6207c1e2-dbda-4fa4-b6a4-195e27245a69 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:17.232733 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f301c0-e9d9-4b4d-b99a-f0280051b230 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:17.233343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f301c0-e9d9-4b4d-b99a-f0280051b230 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:17.234181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f301c0-e9d9-4b4d-b99a-f0280051b230 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:17.234452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f301c0-e9d9-4b4d-b99a-f0280051b230 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:17.241483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f301c0-e9d9-4b4d-b99a-f0280051b230 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:17.242883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f301c0-e9d9-4b4d-b99a-f0280051b230 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-66f301c0-e9d9-4b4d-b99a-f0280051b230 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 28 12:05:17.253442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:17.254518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:17.255665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:17.255665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:17.255665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 28 12:05:17.255665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 28 12:05:17.255665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 28 12:05:17.255665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 28 12:05:17.255665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 28 12:05:17.255665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 28 12:05:17.255665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 28 12:05:17.255665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:17.255665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 649/1280] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:05:17 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:17.260490 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d76b7d-07fa-489c-938e-e1e3cc742122 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:17.264404 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:17.356093 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d76b7d-07fa-489c-938e-e1e3cc742122 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:17.388525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69d5add-e141-4dee-b849-05cc75126eed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:17.389117 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69d5add-e141-4dee-b849-05cc75126eed None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:17.389394 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69d5add-e141-4dee-b849-05cc75126eed None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:17.390172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69d5add-e141-4dee-b849-05cc75126eed None admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:17.390893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69d5add-e141-4dee-b849-05cc75126eed None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:17.391196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69d5add-e141-4dee-b849-05cc75126eed None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:17.392645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69d5add-e141-4dee-b849-05cc75126eed None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:17.409403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 632/1281] 10.222.0.26 () {56 vars in 1143 bytes} [Wed Sep 28 12:05:17 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:17.415264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f301c0-e9d9-4b4d-b99a-f0280051b230 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:17.500617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:17.510141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f301c0-e9d9-4b4d-b99a-f0280051b230 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:17.542507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59171de7-28f4-4aee-b165-3dc3c75710b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:17.543091 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59171de7-28f4-4aee-b165-3dc3c75710b4 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:17.543364 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59171de7-28f4-4aee-b165-3dc3c75710b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:17.543630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59171de7-28f4-4aee-b165-3dc3c75710b4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:17.607946 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59171de7-28f4-4aee-b165-3dc3c75710b4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:17.609353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59171de7-28f4-4aee-b165-3dc3c75710b4 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:17.691708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:17.701725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 650/1282] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Sep 28 12:05:17 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 287 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 28 12:05:17.708041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69d5add-e141-4dee-b849-05cc75126eed None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:17.767461 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:17.804915 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69d5add-e141-4dee-b849-05cc75126eed None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:17.838482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b96ba34-b531-4d75-bb31-2e4b1631c782 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:17.839098 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b96ba34-b531-4d75-bb31-2e4b1631c782 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:17.839378 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b96ba34-b531-4d75-bb31-2e4b1631c782 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:17.839916 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b96ba34-b531-4d75-bb31-2e4b1631c782 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:17.849702 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b96ba34-b531-4d75-bb31-2e4b1631c782 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:17.851143 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b96ba34-b531-4d75-bb31-2e4b1631c782 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-1b96ba34-b531-4d75-bb31-2e4b1631c782 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 28 12:05:17.865279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:17.866348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 28 12:05:17.867574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 28 12:05:17.867574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 28 12:05:17.867574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:05:17.867574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 633/1283] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:05:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:17.871983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59171de7-28f4-4aee-b165-3dc3c75710b4 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:17.965709 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59171de7-28f4-4aee-b165-3dc3c75710b4 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:17.997042 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8799439e-9435-49d9-9613-4b44de7f35cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:17.997628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8799439e-9435-49d9-9613-4b44de7f35cb None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:17.997902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8799439e-9435-49d9-9613-4b44de7f35cb None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:17.998169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8799439e-9435-49d9-9613-4b44de7f35cb None admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:17.998796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8799439e-9435-49d9-9613-4b44de7f35cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:17.999634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8799439e-9435-49d9-9613-4b44de7f35cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:18.000983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8799439e-9435-49d9-9613-4b44de7f35cb None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:18.011460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 651/1284] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Sep 28 12:05:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:18.017558 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b96ba34-b531-4d75-bb31-2e4b1631c782 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:18.111632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b96ba34-b531-4d75-bb31-2e4b1631c782 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:18.126072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:18.146233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73ce2f5-7c96-4a83-8d76-5c112b4e45d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:18.146859 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73ce2f5-7c96-4a83-8d76-5c112b4e45d6 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:18.147135 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73ce2f5-7c96-4a83-8d76-5c112b4e45d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:18.147407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73ce2f5-7c96-4a83-8d76-5c112b4e45d6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:18.153828 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73ce2f5-7c96-4a83-8d76-5c112b4e45d6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:18.155288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73ce2f5-7c96-4a83-8d76-5c112b4e45d6 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application [None req-f73ce2f5-7c96-4a83-8d76-5c112b4e45d6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 28 12:05:18.166919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:18.168021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:18.169305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:05:18.169305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:05:18.169305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:05:18.169305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:05:18.169305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:05:18.169305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:05:18.169305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 28 12:05:18.169305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: ERROR keystone.server.flask.application Sep 28 12:05:18.169305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 634/1285] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Sep 28 12:05:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:18.174423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8799439e-9435-49d9-9613-4b44de7f35cb None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:18.222644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:18.270247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8799439e-9435-49d9-9613-4b44de7f35cb None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:18.301632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fa6396-0ac9-4301-b26f-94bc26c51b94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:18.302208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa6396-0ac9-4301-b26f-94bc26c51b94 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:18.302481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa6396-0ac9-4301-b26f-94bc26c51b94 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:18.302747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa6396-0ac9-4301-b26f-94bc26c51b94 None admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:18.303420 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fa6396-0ac9-4301-b26f-94bc26c51b94 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:18.303718 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fa6396-0ac9-4301-b26f-94bc26c51b94 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:18.305145 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fa6396-0ac9-4301-b26f-94bc26c51b94 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:18.317023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 652/1286] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Sep 28 12:05:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:18.323085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73ce2f5-7c96-4a83-8d76-5c112b4e45d6 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:18.418597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73ce2f5-7c96-4a83-8d76-5c112b4e45d6 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:18.451229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e7a666-e071-44ae-ba20-3882316ed7a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:18.451810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e7a666-e071-44ae-ba20-3882316ed7a8 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:18.452084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e7a666-e071-44ae-ba20-3882316ed7a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:18.452418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e7a666-e071-44ae-ba20-3882316ed7a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:18.453129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e7a666-e071-44ae-ba20-3882316ed7a8 None admin] RBAC: Adding query filter params (role.id=f80bb7fe38494b9aabf015d2ff6fa215, scope.project.id=8695d562e3e24c7c9300cf3a23cd7502, user.id=3758ea0f26264340ad2f154c0111af21) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:18.453430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e7a666-e071-44ae-ba20-3882316ed7a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:18.454843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e7a666-e071-44ae-ba20-3882316ed7a8 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:18.460350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 635/1287] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Sep 28 12:05:18 2022] GET /identity/v3/role_assignments?user.id=3758ea0f26264340ad2f154c0111af21&scope.project.id=8695d562e3e24c7c9300cf3a23cd7502&role.id=f80bb7fe38494b9aabf015d2ff6fa215 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:20.600175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e87be35-7514-41ac-b23e-cbbd31bf00ca None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:20.600741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e87be35-7514-41ac-b23e-cbbd31bf00ca None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:20.600741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e87be35-7514-41ac-b23e-cbbd31bf00ca None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:20.602074 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 653/1288] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:05:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:05:20.607270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd6b95d-a642-4634-bd6b-1611aba33cf5 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:20.607703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd6b95d-a642-4634-bd6b-1611aba33cf5 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:20.607841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd6b95d-a642-4634-bd6b-1611aba33cf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:20.626776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-cbd6b95d-a642-4634-bd6b-1611aba33cf5 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:05:20.627963 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-cbd6b95d-a642-4634-bd6b-1611aba33cf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:05:20.655877 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:20.692191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 636/1289] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:05:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:05:20.960778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e87be35-7514-41ac-b23e-cbbd31bf00ca None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:21.057157 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e87be35-7514-41ac-b23e-cbbd31bf00ca None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:21.067143 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:21.090159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e7fa7c-17cd-4af0-8faa-7b453086b04f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:21.090768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e7fa7c-17cd-4af0-8faa-7b453086b04f None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:21.091046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e7fa7c-17cd-4af0-8faa-7b453086b04f None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:21.091321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e7fa7c-17cd-4af0-8faa-7b453086b04f None admin] PATH_INFO: `/v3/services/placement` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:21.091898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e7fa7c-17cd-4af0-8faa-7b453086b04f None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:21.093794 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e7fa7c-17cd-4af0-8faa-7b453086b04f None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-d5e7fa7c-17cd-4af0-8faa-7b453086b04f None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:21.102206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:21.103405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:21.104239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 654/1290] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Sep 28 12:05:20 2022] GET /identity/v3/services/placement => generated 90 bytes in 143 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:21.109659 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd6b95d-a642-4634-bd6b-1611aba33cf5 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:21.212838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd6b95d-a642-4634-bd6b-1611aba33cf5 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:21.246232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f979fd4-394d-4866-abe1-9be4a9b9f61b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:21.246835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f979fd4-394d-4866-abe1-9be4a9b9f61b None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:21.247120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f979fd4-394d-4866-abe1-9be4a9b9f61b None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:21.247392 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f979fd4-394d-4866-abe1-9be4a9b9f61b None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:21.248022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f979fd4-394d-4866-abe1-9be4a9b9f61b None admin] RBAC: Adding query filter params (name=placement) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:21.248347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f979fd4-394d-4866-abe1-9be4a9b9f61b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:21.249747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f979fd4-394d-4866-abe1-9be4a9b9f61b None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:21.254415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 637/1291] 10.222.0.26 () {56 vars in 1116 bytes} [Wed Sep 28 12:05:21 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:21.260278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e7fa7c-17cd-4af0-8faa-7b453086b04f None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:21.356211 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e7fa7c-17cd-4af0-8faa-7b453086b04f None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:21.387873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a57c3cc-5800-46c2-8272-9a301c4e1c5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:21.388484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a57c3cc-5800-46c2-8272-9a301c4e1c5a None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:21.388767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a57c3cc-5800-46c2-8272-9a301c4e1c5a None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:21.389040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a57c3cc-5800-46c2-8272-9a301c4e1c5a None admin] PATH_INFO: `/v3/services` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:21.390190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a57c3cc-5800-46c2-8272-9a301c4e1c5a None admin] RBAC: Adding query filter params (type=placement) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:21.390496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a57c3cc-5800-46c2-8272-9a301c4e1c5a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:21.391851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a57c3cc-5800-46c2-8272-9a301c4e1c5a None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:21.396069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 655/1292] 10.222.0.26 () {56 vars in 1116 bytes} [Wed Sep 28 12:05:21 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:23.515184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aa3a7c-e8f8-47df-aa5d-a4ede5f460eb None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:23.515616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aa3a7c-e8f8-47df-aa5d-a4ede5f460eb None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:23.515708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aa3a7c-e8f8-47df-aa5d-a4ede5f460eb None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:23.517090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 638/1293] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:05:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:05:23.522295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c6041c-f439-4d67-9242-61902fb32141 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:23.522696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c6041c-f439-4d67-9242-61902fb32141 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:23.522920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c6041c-f439-4d67-9242-61902fb32141 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:23.540607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-52c6041c-f439-4d67-9242-61902fb32141 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:05:23.541789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-52c6041c-f439-4d67-9242-61902fb32141 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:05:23.570681 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:23.606779 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 656/1294] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:05:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:05:23.875370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36aa3a7c-e8f8-47df-aa5d-a4ede5f460eb None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:23.904829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:23.973722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36aa3a7c-e8f8-47df-aa5d-a4ede5f460eb None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:24.006129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e571f039-cc71-447c-935c-d1dd7772acea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:24.006717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e571f039-cc71-447c-935c-d1dd7772acea None admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:24.006996 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e571f039-cc71-447c-935c-d1dd7772acea None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:24.007267 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e571f039-cc71-447c-935c-d1dd7772acea None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:24.007918 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e571f039-cc71-447c-935c-d1dd7772acea None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:24.009317 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e571f039-cc71-447c-935c-d1dd7772acea None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:24.023524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 639/1295] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Sep 28 12:05:23 2022] POST /identity/v3/services => generated 246 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:05:26.151030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d00390e-1be5-4bbc-8b2d-ce988c4b44e2 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:26.151571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d00390e-1be5-4bbc-8b2d-ce988c4b44e2 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:26.151571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d00390e-1be5-4bbc-8b2d-ce988c4b44e2 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:26.152931 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 657/1296] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:05:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 28 12:05:26.160121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cb22e0-474d-4c0d-9dba-b9dc910c9e96 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:26.160628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cb22e0-474d-4c0d-9dba-b9dc910c9e96 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:26.160932 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cb22e0-474d-4c0d-9dba-b9dc910c9e96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:26.185718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-e1cb22e0-474d-4c0d-9dba-b9dc910c9e96 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:05:26.186928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e1cb22e0-474d-4c0d-9dba-b9dc910c9e96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:05:26.261147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 640/1297] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:05:26 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:05:26.524604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d00390e-1be5-4bbc-8b2d-ce988c4b44e2 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:26.620284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d00390e-1be5-4bbc-8b2d-ce988c4b44e2 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:26.652129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b947f632-cb5d-45fa-9727-d4edd90de5c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:26.652758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b947f632-cb5d-45fa-9727-d4edd90de5c5 None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:26.653030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b947f632-cb5d-45fa-9727-d4edd90de5c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:26.653296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b947f632-cb5d-45fa-9727-d4edd90de5c5 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:26.653858 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b947f632-cb5d-45fa-9727-d4edd90de5c5 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:26.655239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b947f632-cb5d-45fa-9727-d4edd90de5c5 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-b947f632-cb5d-45fa-9727-d4edd90de5c5 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:26.663552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:26.664736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:26.665844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 658/1298] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Sep 28 12:05:26 2022] GET /identity/v3/services/placement => generated 90 bytes in 141 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:26.670322 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cb22e0-474d-4c0d-9dba-b9dc910c9e96 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:26.688356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:26.769400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cb22e0-474d-4c0d-9dba-b9dc910c9e96 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:26.803987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3719d992-c180-417d-a267-4733051d79e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:26.804615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3719d992-c180-417d-a267-4733051d79e6 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:26.804895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3719d992-c180-417d-a267-4733051d79e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:26.805165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3719d992-c180-417d-a267-4733051d79e6 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:26.805788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3719d992-c180-417d-a267-4733051d79e6 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:26.806089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3719d992-c180-417d-a267-4733051d79e6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:26.807449 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3719d992-c180-417d-a267-4733051d79e6 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:26.812991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 641/1299] 10.222.0.26 () {56 vars in 1116 bytes} [Wed Sep 28 12:05:26 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:26.818933 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b947f632-cb5d-45fa-9727-d4edd90de5c5 None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:26.850391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:26.915468 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b947f632-cb5d-45fa-9727-d4edd90de5c5 None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:26.946914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7b652a-73bb-48ef-bfda-a25f468e3eea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:26.947492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7b652a-73bb-48ef-bfda-a25f468e3eea None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:26.947766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7b652a-73bb-48ef-bfda-a25f468e3eea None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:26.948035 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7b652a-73bb-48ef-bfda-a25f468e3eea None admin] PATH_INFO: `/v3/endpoints` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:26.948749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7b652a-73bb-48ef-bfda-a25f468e3eea None admin] RBAC: Adding query filter params (interface=public, service_id=4c92f3348d064161b0c86744c88f40a4, region_id=RegionOne) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:26.949053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7b652a-73bb-48ef-bfda-a25f468e3eea None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:26.950386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7b652a-73bb-48ef-bfda-a25f468e3eea None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:26.955746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 659/1300] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Sep 28 12:05:26 2022] GET /identity/v3/endpoints?service_id=4c92f3348d064161b0c86744c88f40a4&interface=public®ion_id=RegionOne => generated 194 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:26.961416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3719d992-c180-417d-a267-4733051d79e6 None admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:27.039849 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:27.057973 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3719d992-c180-417d-a267-4733051d79e6 None admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:27.090231 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61490c30-2c50-42c7-bd4c-4f2234ea651c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:27.090842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61490c30-2c50-42c7-bd4c-4f2234ea651c None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:27.091119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61490c30-2c50-42c7-bd4c-4f2234ea651c None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:27.091391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61490c30-2c50-42c7-bd4c-4f2234ea651c None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:27.092008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61490c30-2c50-42c7-bd4c-4f2234ea651c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:27.093932 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61490c30-2c50-42c7-bd4c-4f2234ea651c None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:27.101300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 642/1301] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Sep 28 12:05:26 2022] GET /identity/v3/services => generated 2191 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:05:29.221409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22713422-0a23-43a6-864a-05025bf80da3 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:29.221952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22713422-0a23-43a6-864a-05025bf80da3 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:29.221952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22713422-0a23-43a6-864a-05025bf80da3 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:29.223292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 660/1302] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:05:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:05:29.228443 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8359ac66-9a78-4a9c-ac6f-9e7bd61519e3 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:29.228855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8359ac66-9a78-4a9c-ac6f-9e7bd61519e3 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:29.228985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8359ac66-9a78-4a9c-ac6f-9e7bd61519e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:29.247296 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-8359ac66-9a78-4a9c-ac6f-9e7bd61519e3 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:05:29.248501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8359ac66-9a78-4a9c-ac6f-9e7bd61519e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:05:29.312227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 643/1303] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:05:29 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 28 12:05:29.578400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22713422-0a23-43a6-864a-05025bf80da3 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:29.670087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:29.677590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22713422-0a23-43a6-864a-05025bf80da3 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:29.710010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9b44bd-e76e-4923-9922-51d0cae258ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:29.710616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b44bd-e76e-4923-9922-51d0cae258ea None admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:29.710893 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b44bd-e76e-4923-9922-51d0cae258ea None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:29.711166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b44bd-e76e-4923-9922-51d0cae258ea None admin] PATH_INFO: `/v3/services/placement` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:29.711739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9b44bd-e76e-4923-9922-51d0cae258ea None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:29.713110 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9b44bd-e76e-4923-9922-51d0cae258ea None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-af9b44bd-e76e-4923-9922-51d0cae258ea None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:05:29.721784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:05:29.722885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:05:29.723713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 661/1304] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Sep 28 12:05:29 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 28 12:05:29.728921 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8359ac66-9a78-4a9c-ac6f-9e7bd61519e3 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:29.803958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:29.826536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8359ac66-9a78-4a9c-ac6f-9e7bd61519e3 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:29.860159 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8222d538-3a7d-4511-b092-9be533c4f0c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:29.860798 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8222d538-3a7d-4511-b092-9be533c4f0c7 None admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:29.861078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8222d538-3a7d-4511-b092-9be533c4f0c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:29.861354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8222d538-3a7d-4511-b092-9be533c4f0c7 None admin] PATH_INFO: `/v3/services` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:29.861975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8222d538-3a7d-4511-b092-9be533c4f0c7 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:05:29.862276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8222d538-3a7d-4511-b092-9be533c4f0c7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:29.863635 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8222d538-3a7d-4511-b092-9be533c4f0c7 None admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:29.868599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 644/1305] 10.222.0.26 () {56 vars in 1116 bytes} [Wed Sep 28 12:05:29 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:05:29.874376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9b44bd-e76e-4923-9922-51d0cae258ea None admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:05:29.969567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9b44bd-e76e-4923-9922-51d0cae258ea None admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:05:30.001357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6278a4fd-fa00-4d3f-a096-034e04920190 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:05:30.001939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6278a4fd-fa00-4d3f-a096-034e04920190 None admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:05:30.002216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6278a4fd-fa00-4d3f-a096-034e04920190 None admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:05:30.002486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6278a4fd-fa00-4d3f-a096-034e04920190 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:05:30.003134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6278a4fd-fa00-4d3f-a096-034e04920190 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:05:30.004506 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6278a4fd-fa00-4d3f-a096-034e04920190 None admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:05:30.023186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 662/1306] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:05:29 2022] POST /identity/v3/endpoints => generated 331 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:05:39.865761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=30 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:05:40.019009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:33.901119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd0f352-0661-4766-ab41-c489c344b406 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:33.901574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd0f352-0661-4766-ab41-c489c344b406 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:33.901574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd0f352-0661-4766-ab41-c489c344b406 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:33.906901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 645/1307] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:06:33 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:06:33.908290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d65705-a4f3-4fe5-a22f-4117b1f6fe06 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:33.908734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d65705-a4f3-4fe5-a22f-4117b1f6fe06 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:33.908972 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d65705-a4f3-4fe5-a22f-4117b1f6fe06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:34.029544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-c4d65705-a4f3-4fe5-a22f-4117b1f6fe06 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:06:34.031214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c4d65705-a4f3-4fe5-a22f-4117b1f6fe06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:34.146699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 663/1308] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:06:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:06:34.448537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0af529e-12cb-4dd2-b983-ee6b61800ce8 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:34.448985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0af529e-12cb-4dd2-b983-ee6b61800ce8 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:34.449362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0af529e-12cb-4dd2-b983-ee6b61800ce8 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:34.451344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 646/1309] 10.222.0.26 () {54 vars in 964 bytes} [Wed Sep 28 12:06:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:06:34.457851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1948a100-98ad-4a6e-b1a8-2c3f7fac309d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:34.458138 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1948a100-98ad-4a6e-b1a8-2c3f7fac309d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:34.458407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1948a100-98ad-4a6e-b1a8-2c3f7fac309d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:34.561816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-1948a100-98ad-4a6e-b1a8-2c3f7fac309d None None] MFA Rules not processed for user `0e429f061bd04c28b0c60c593b64fb8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:06:34.563146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1948a100-98ad-4a6e-b1a8-2c3f7fac309d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:34.586354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:34.653004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 664/1310] 10.222.0.26 () {58 vars in 1061 bytes} [Wed Sep 28 12:06:34 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:06:34.661987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0af529e-12cb-4dd2-b983-ee6b61800ce8 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:06:34.664018 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e0af529e-12cb-4dd2-b983-ee6b61800ce8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:34.779261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0af529e-12cb-4dd2-b983-ee6b61800ce8 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:06:34.856002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d258d7-8723-4ebf-94e9-51bdf20d281a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:06:34.856651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d258d7-8723-4ebf-94e9-51bdf20d281a service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:34.857464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d258d7-8723-4ebf-94e9-51bdf20d281a service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:34.857741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d258d7-8723-4ebf-94e9-51bdf20d281a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:34.859329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e5d258d7-8723-4ebf-94e9-51bdf20d281a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:34.916827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:34.974767 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d258d7-8723-4ebf-94e9-51bdf20d281a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:06:34.976230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d258d7-8723-4ebf-94e9-51bdf20d281a service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:06:34.995927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 647/1311] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:06:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:06:37.518647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0014f1f9-faf5-4cce-824e-42ba75339089 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:37.519127 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0014f1f9-faf5-4cce-824e-42ba75339089 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:37.519237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0014f1f9-faf5-4cce-824e-42ba75339089 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:37.520608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 665/1312] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:06:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:06:37.525810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ea01e6-273a-434f-a9e8-1a4209bf95da None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:37.526250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ea01e6-273a-434f-a9e8-1a4209bf95da None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:37.526353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ea01e6-273a-434f-a9e8-1a4209bf95da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:37.595872 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-d1ea01e6-273a-434f-a9e8-1a4209bf95da None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:06:37.597036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d1ea01e6-273a-434f-a9e8-1a4209bf95da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:37.690639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 648/1313] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:06:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:06:37.967523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0014f1f9-faf5-4cce-824e-42ba75339089 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:06:37.969391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0014f1f9-faf5-4cce-824e-42ba75339089 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:38.071093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0014f1f9-faf5-4cce-824e-42ba75339089 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:06:38.141640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:38.144209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef76bb96-625d-4b01-ae07-baa0d325b666 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:06:38.145006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef76bb96-625d-4b01-ae07-baa0d325b666 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:38.145332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef76bb96-625d-4b01-ae07-baa0d325b666 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:38.145742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef76bb96-625d-4b01-ae07-baa0d325b666 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:38.147561 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ef76bb96-625d-4b01-ae07-baa0d325b666 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:38.252552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef76bb96-625d-4b01-ae07-baa0d325b666 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:06:38.253965 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef76bb96-625d-4b01-ae07-baa0d325b666 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:06:38.271163 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 666/1314] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:06:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:06:40.682454 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e34531-a083-4104-89ba-ead012f10804 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:40.683012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e34531-a083-4104-89ba-ead012f10804 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:40.683012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e34531-a083-4104-89ba-ead012f10804 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:40.684356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 649/1315] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:06:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:06:40.689673 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6a9bd3-70c2-4ea9-8600-67ac83156526 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:40.690098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6a9bd3-70c2-4ea9-8600-67ac83156526 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:40.690250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6a9bd3-70c2-4ea9-8600-67ac83156526 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:40.754543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-7a6a9bd3-70c2-4ea9-8600-67ac83156526 None None] MFA Rules not processed for user `e481e627a19f4fcaa9af5c5635ce4374`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:06:40.755655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7a6a9bd3-70c2-4ea9-8600-67ac83156526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:40.805324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:40.849132 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 667/1316] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:06:40 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:06:41.128629 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e34531-a083-4104-89ba-ead012f10804 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:06:41.130167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-18e34531-a083-4104-89ba-ead012f10804 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:41.140447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:41.238283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e34531-a083-4104-89ba-ead012f10804 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:06:41.313268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c0bdcd-b3a1-436c-81bf-ffe676bac97f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:06:41.313869 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c0bdcd-b3a1-436c-81bf-ffe676bac97f service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:41.314147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c0bdcd-b3a1-436c-81bf-ffe676bac97f service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:41.314420 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c0bdcd-b3a1-436c-81bf-ffe676bac97f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:41.316003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-59c0bdcd-b3a1-436c-81bf-ffe676bac97f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:41.377628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:41.423186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c0bdcd-b3a1-436c-81bf-ffe676bac97f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:06:41.424681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c0bdcd-b3a1-436c-81bf-ffe676bac97f service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:06:41.442328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 650/1317] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:06:41 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:06:44.123049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cfcf40-cd9e-4776-9cee-1eb364fd43e0 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:44.123613 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cfcf40-cd9e-4776-9cee-1eb364fd43e0 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:44.123613 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cfcf40-cd9e-4776-9cee-1eb364fd43e0 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:44.124954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 668/1318] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:06:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:06:44.130316 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99033a8-61fd-402b-ae03-85ef78a60931 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:44.130749 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99033a8-61fd-402b-ae03-85ef78a60931 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:44.130868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99033a8-61fd-402b-ae03-85ef78a60931 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:44.196582 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-b99033a8-61fd-402b-ae03-85ef78a60931 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:06:44.197783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b99033a8-61fd-402b-ae03-85ef78a60931 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:44.289986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 651/1319] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:06:44 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:06:44.594874 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f711d1-3cc2-4ea7-ba7c-206f2b208601 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:44.595160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f711d1-3cc2-4ea7-ba7c-206f2b208601 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:44.595435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f711d1-3cc2-4ea7-ba7c-206f2b208601 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:44.596810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 669/1320] 10.222.0.26 () {54 vars in 964 bytes} [Wed Sep 28 12:06:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:06:44.602567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d131b-8707-419f-9117-459fb50c6d59 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:44.602842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d131b-8707-419f-9117-459fb50c6d59 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:44.603118 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d131b-8707-419f-9117-459fb50c6d59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:44.624945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:44.678331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-d88d131b-8707-419f-9117-459fb50c6d59 None None] MFA Rules not processed for user `0e429f061bd04c28b0c60c593b64fb8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:06:44.679535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d88d131b-8707-419f-9117-459fb50c6d59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:44.753668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 652/1321] 10.222.0.26 () {58 vars in 1061 bytes} [Wed Sep 28 12:06:44 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:06:44.762584 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f711d1-3cc2-4ea7-ba7c-206f2b208601 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:06:44.764747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-61f711d1-3cc2-4ea7-ba7c-206f2b208601 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:44.864157 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f711d1-3cc2-4ea7-ba7c-206f2b208601 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:06:44.936809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57096a80-6b35-4303-a2ed-04c80d22e7b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:06:44.937423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57096a80-6b35-4303-a2ed-04c80d22e7b4 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:44.937696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57096a80-6b35-4303-a2ed-04c80d22e7b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:44.937963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57096a80-6b35-4303-a2ed-04c80d22e7b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:44.939565 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-57096a80-6b35-4303-a2ed-04c80d22e7b4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:44.946444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:45.045398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57096a80-6b35-4303-a2ed-04c80d22e7b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:06:45.046819 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57096a80-6b35-4303-a2ed-04c80d22e7b4 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:06:45.065074 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 670/1322] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:06:44 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:06:48.359914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46b3253-122d-4133-b695-ec7c29e07e7b None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:48.360547 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46b3253-122d-4133-b695-ec7c29e07e7b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:48.360547 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46b3253-122d-4133-b695-ec7c29e07e7b None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:48.361886 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 653/1323] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:06:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:06:48.367758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d64e44-f246-4373-90e0-68586a2bc84e None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:48.368296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d64e44-f246-4373-90e0-68586a2bc84e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:48.368296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d64e44-f246-4373-90e0-68586a2bc84e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:48.439739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-08d64e44-f246-4373-90e0-68586a2bc84e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:06:48.441006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-08d64e44-f246-4373-90e0-68586a2bc84e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:48.502610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:48.544297 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 671/1324] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:06:48.828048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46b3253-122d-4133-b695-ec7c29e07e7b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:06:48.829630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c46b3253-122d-4133-b695-ec7c29e07e7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:48.933554 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46b3253-122d-4133-b695-ec7c29e07e7b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:06:48.946845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:49.010358 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52a4f04-cc74-4965-a1c7-698869fe59ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:06:49.010965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52a4f04-cc74-4965-a1c7-698869fe59ea service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:49.011243 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52a4f04-cc74-4965-a1c7-698869fe59ea service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:49.011515 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52a4f04-cc74-4965-a1c7-698869fe59ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:49.013673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d52a4f04-cc74-4965-a1c7-698869fe59ea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:49.121566 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52a4f04-cc74-4965-a1c7-698869fe59ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:06:49.123042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52a4f04-cc74-4965-a1c7-698869fe59ea service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:06:49.141363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 654/1325] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:06:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:06:51.363935 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b568c598-7e46-4ed6-a9f9-5e4a4a2610d0 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:51.364575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b568c598-7e46-4ed6-a9f9-5e4a4a2610d0 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:51.364575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b568c598-7e46-4ed6-a9f9-5e4a4a2610d0 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:51.365866 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 672/1326] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:06:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:06:51.371199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7c30c7-b8d6-4674-b0c0-3d595195a7ee None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:51.371583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7c30c7-b8d6-4674-b0c0-3d595195a7ee None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:51.371750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7c30c7-b8d6-4674-b0c0-3d595195a7ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:51.428919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:51.448902 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-2a7c30c7-b8d6-4674-b0c0-3d595195a7ee None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:06:51.450107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2a7c30c7-b8d6-4674-b0c0-3d595195a7ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:51.550292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 655/1327] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:06:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:06:51.830114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b568c598-7e46-4ed6-a9f9-5e4a4a2610d0 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:06:51.831721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b568c598-7e46-4ed6-a9f9-5e4a4a2610d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:51.935245 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b568c598-7e46-4ed6-a9f9-5e4a4a2610d0 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:06:52.016705 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a861c839-9ebf-4745-9a29-cff59a5cf8c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:06:52.017320 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a861c839-9ebf-4745-9a29-cff59a5cf8c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:52.017600 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a861c839-9ebf-4745-9a29-cff59a5cf8c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:52.017875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a861c839-9ebf-4745-9a29-cff59a5cf8c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:52.019487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a861c839-9ebf-4745-9a29-cff59a5cf8c8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:52.038926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:52.131861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a861c839-9ebf-4745-9a29-cff59a5cf8c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:06:52.133261 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a861c839-9ebf-4745-9a29-cff59a5cf8c8 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:06:52.151576 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 673/1328] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:06:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:06:54.345765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7c3755-67bf-4478-a936-33c0b86c7542 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:54.346292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7c3755-67bf-4478-a936-33c0b86c7542 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:54.346292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7c3755-67bf-4478-a936-33c0b86c7542 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:54.347639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 656/1329] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:06:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:06:54.353042 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c103d0-2815-4601-834d-1b7577e6cd22 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:54.353505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c103d0-2815-4601-834d-1b7577e6cd22 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:54.353629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c103d0-2815-4601-834d-1b7577e6cd22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:54.421519 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-69c103d0-2815-4601-834d-1b7577e6cd22 None None] MFA Rules not processed for user `e481e627a19f4fcaa9af5c5635ce4374`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:06:54.422646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-69c103d0-2815-4601-834d-1b7577e6cd22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:54.486622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:54.519705 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 674/1330] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:06:54 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:06:54.805118 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7c3755-67bf-4478-a936-33c0b86c7542 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:06:54.806688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9f7c3755-67bf-4478-a936-33c0b86c7542 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:54.907815 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7c3755-67bf-4478-a936-33c0b86c7542 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:06:54.929318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:54.984397 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13073f28-8e61-477f-b8fe-ef3ce2edb724 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:06:54.984903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13073f28-8e61-477f-b8fe-ef3ce2edb724 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:54.985179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13073f28-8e61-477f-b8fe-ef3ce2edb724 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:54.985444 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13073f28-8e61-477f-b8fe-ef3ce2edb724 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:54.987051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-13073f28-8e61-477f-b8fe-ef3ce2edb724 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:55.095978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13073f28-8e61-477f-b8fe-ef3ce2edb724 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:06:55.097381 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13073f28-8e61-477f-b8fe-ef3ce2edb724 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:06:55.115225 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 657/1331] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:06:54 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:06:57.549218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b71449-421f-4e63-9bf5-80d0acdefb56 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:57.549720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b71449-421f-4e63-9bf5-80d0acdefb56 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:57.549720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b71449-421f-4e63-9bf5-80d0acdefb56 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:57.551019 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 675/1332] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:06:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:06:57.556198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c2dfdb-babc-4169-9d6d-21a18b027e77 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:57.556620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c2dfdb-babc-4169-9d6d-21a18b027e77 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:57.556788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c2dfdb-babc-4169-9d6d-21a18b027e77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:57.619081 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:57.621949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-65c2dfdb-babc-4169-9d6d-21a18b027e77 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:06:57.624163 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-65c2dfdb-babc-4169-9d6d-21a18b027e77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:57.722523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 658/1333] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:06:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:06:58.002346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b71449-421f-4e63-9bf5-80d0acdefb56 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:06:58.003883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-49b71449-421f-4e63-9bf5-80d0acdefb56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:58.105550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b71449-421f-4e63-9bf5-80d0acdefb56 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:06:58.177583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1281845d-076e-42e7-82ed-baae8e0e38e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:06:58.178179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1281845d-076e-42e7-82ed-baae8e0e38e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:06:58.178455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1281845d-076e-42e7-82ed-baae8e0e38e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:06:58.178723 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1281845d-076e-42e7-82ed-baae8e0e38e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:06:58.180301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1281845d-076e-42e7-82ed-baae8e0e38e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:06:58.206939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:06:58.287799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1281845d-076e-42e7-82ed-baae8e0e38e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:06:58.289248 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1281845d-076e-42e7-82ed-baae8e0e38e9 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:06:58.307024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 676/1334] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:06:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:00.488717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd1810-d44e-46b0-a84e-472ca32b8084 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:00.489252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd1810-d44e-46b0-a84e-472ca32b8084 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:00.489252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd1810-d44e-46b0-a84e-472ca32b8084 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:00.490592 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 659/1335] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:00.496361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deca8d84-a711-466b-9fb2-311b7572cdfe None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:00.496745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deca8d84-a711-466b-9fb2-311b7572cdfe None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:00.496918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deca8d84-a711-466b-9fb2-311b7572cdfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:00.565788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-deca8d84-a711-466b-9fb2-311b7572cdfe None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:00.566982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-deca8d84-a711-466b-9fb2-311b7572cdfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:00.639820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:00.661759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 677/1336] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:07:00.945649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cd1810-d44e-46b0-a84e-472ca32b8084 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:00.947185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b5cd1810-d44e-46b0-a84e-472ca32b8084 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:01.061348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cd1810-d44e-46b0-a84e-472ca32b8084 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:01.090670 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:01.139521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcd37da-60c5-4299-ad8c-88153243d38f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:01.140116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd37da-60c5-4299-ad8c-88153243d38f service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:01.140423 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd37da-60c5-4299-ad8c-88153243d38f service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:01.140692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd37da-60c5-4299-ad8c-88153243d38f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:01.142392 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5fcd37da-60c5-4299-ad8c-88153243d38f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:01.255219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcd37da-60c5-4299-ad8c-88153243d38f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:01.257453 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcd37da-60c5-4299-ad8c-88153243d38f service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:01.276208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 660/1337] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:03.789915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678d2a7b-c749-4f85-a5a4-9823e7b22ec1 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:03.790435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678d2a7b-c749-4f85-a5a4-9823e7b22ec1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:03.790530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678d2a7b-c749-4f85-a5a4-9823e7b22ec1 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:03.791807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 678/1338] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:03.797432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3672fe2a-1b44-4922-a474-c9a0d1257815 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:03.797859 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3672fe2a-1b44-4922-a474-c9a0d1257815 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:03.797954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3672fe2a-1b44-4922-a474-c9a0d1257815 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:03.861366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3672fe2a-1b44-4922-a474-c9a0d1257815 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:03.862550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3672fe2a-1b44-4922-a474-c9a0d1257815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:03.870521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:03.959267 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 661/1339] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:03 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:04.239318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678d2a7b-c749-4f85-a5a4-9823e7b22ec1 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:04.240862 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-678d2a7b-c749-4f85-a5a4-9823e7b22ec1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:04.340595 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678d2a7b-c749-4f85-a5a4-9823e7b22ec1 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:04.412445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5be0cab-96b2-476a-bf55-db6efc877d98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:04.413053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5be0cab-96b2-476a-bf55-db6efc877d98 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:04.413326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5be0cab-96b2-476a-bf55-db6efc877d98 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:04.413590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5be0cab-96b2-476a-bf55-db6efc877d98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:04.415132 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b5be0cab-96b2-476a-bf55-db6efc877d98 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:04.449228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:04.522930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5be0cab-96b2-476a-bf55-db6efc877d98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:04.524424 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5be0cab-96b2-476a-bf55-db6efc877d98 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:04.542376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 679/1340] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:04 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:08.594631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f753493-9449-4dc8-b30a-97e19f1241fb None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:08.595222 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f753493-9449-4dc8-b30a-97e19f1241fb None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:08.595222 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f753493-9449-4dc8-b30a-97e19f1241fb None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:08.596718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 662/1341] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:08.601975 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53adf9cd-851e-4948-80b5-05879c7b788c None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:08.602428 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53adf9cd-851e-4948-80b5-05879c7b788c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:08.602526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53adf9cd-851e-4948-80b5-05879c7b788c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:08.673162 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-53adf9cd-851e-4948-80b5-05879c7b788c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:08.674523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-53adf9cd-851e-4948-80b5-05879c7b788c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:08.767020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:08.780236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 680/1342] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:09.062716 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f753493-9449-4dc8-b30a-97e19f1241fb None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:09.064296 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8f753493-9449-4dc8-b30a-97e19f1241fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:09.169723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f753493-9449-4dc8-b30a-97e19f1241fb None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:09.207391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:09.243991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa5662c-c102-424f-aa7b-68bc9bdc6120 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:09.244628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa5662c-c102-424f-aa7b-68bc9bdc6120 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:09.244915 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa5662c-c102-424f-aa7b-68bc9bdc6120 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:09.245183 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa5662c-c102-424f-aa7b-68bc9bdc6120 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:09.246758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-efa5662c-c102-424f-aa7b-68bc9bdc6120 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:09.356189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa5662c-c102-424f-aa7b-68bc9bdc6120 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:09.358410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa5662c-c102-424f-aa7b-68bc9bdc6120 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:09.378058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 663/1343] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:12.100644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d89e0d-78fd-4c79-9b27-de7503f3fed2 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:12.101210 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d89e0d-78fd-4c79-9b27-de7503f3fed2 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:12.101210 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d89e0d-78fd-4c79-9b27-de7503f3fed2 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:12.102519 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 681/1344] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:12.108321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acf258c-3944-47ce-9415-a0d2d14124e5 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:12.108738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acf258c-3944-47ce-9415-a0d2d14124e5 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:12.108988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acf258c-3944-47ce-9415-a0d2d14124e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:12.175438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3acf258c-3944-47ce-9415-a0d2d14124e5 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:12.176621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3acf258c-3944-47ce-9415-a0d2d14124e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:12.197001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:12.277783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 664/1345] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:12 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:07:12.559772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d89e0d-78fd-4c79-9b27-de7503f3fed2 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:12.561387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-19d89e0d-78fd-4c79-9b27-de7503f3fed2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:12.668490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d89e0d-78fd-4c79-9b27-de7503f3fed2 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:12.744967 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4923ea2-4a66-4140-9419-29c9d711254f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:12.745591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4923ea2-4a66-4140-9419-29c9d711254f service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:12.745869 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4923ea2-4a66-4140-9419-29c9d711254f service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:12.746141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4923ea2-4a66-4140-9419-29c9d711254f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:12.748378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f4923ea2-4a66-4140-9419-29c9d711254f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:12.794505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:12.863400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4923ea2-4a66-4140-9419-29c9d711254f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:12.864874 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4923ea2-4a66-4140-9419-29c9d711254f service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:12.883914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 682/1346] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:12 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:17.066591 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e90bfe-a82e-4747-829f-f9d1c46a5eb9 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:17.067027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e90bfe-a82e-4747-829f-f9d1c46a5eb9 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:17.067139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e90bfe-a82e-4747-829f-f9d1c46a5eb9 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:17.068532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 665/1347] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:17.073851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28181ed8-8966-4986-8149-b2fee7b60948 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:17.074232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28181ed8-8966-4986-8149-b2fee7b60948 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:17.074432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28181ed8-8966-4986-8149-b2fee7b60948 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:17.140398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-28181ed8-8966-4986-8149-b2fee7b60948 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:17.141562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-28181ed8-8966-4986-8149-b2fee7b60948 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:17.234894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:17.237143 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 683/1348] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:17.518151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e90bfe-a82e-4747-829f-f9d1c46a5eb9 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:17.519737 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-06e90bfe-a82e-4747-829f-f9d1c46a5eb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:17.628141 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e90bfe-a82e-4747-829f-f9d1c46a5eb9 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:17.675319 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:17.701538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a841da-d8fd-4139-8d29-5fd46bef4d63 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:17.702105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a841da-d8fd-4139-8d29-5fd46bef4d63 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:17.702384 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a841da-d8fd-4139-8d29-5fd46bef4d63 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:17.702655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a841da-d8fd-4139-8d29-5fd46bef4d63 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:17.704237 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d7a841da-d8fd-4139-8d29-5fd46bef4d63 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:17.822161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a841da-d8fd-4139-8d29-5fd46bef4d63 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:17.823615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a841da-d8fd-4139-8d29-5fd46bef4d63 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:17.842055 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 666/1349] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:22.718306 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125a923e-f1c0-4ba8-a0df-a2f9d01e608f None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:22.718767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125a923e-f1c0-4ba8-a0df-a2f9d01e608f None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:22.718891 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125a923e-f1c0-4ba8-a0df-a2f9d01e608f None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:22.720400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 684/1350] 10.222.0.26 () {54 vars in 921 bytes} [Wed Sep 28 12:07:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:22.728384 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c137d81b-373c-481e-8462-113be5cf5bcb None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:22.728826 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c137d81b-373c-481e-8462-113be5cf5bcb None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:22.729140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c137d81b-373c-481e-8462-113be5cf5bcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:22.855103 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-c137d81b-373c-481e-8462-113be5cf5bcb None None] MFA Rules not processed for user `3758ea0f26264340ad2f154c0111af21`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:22.856959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c137d81b-373c-481e-8462-113be5cf5bcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:22.878582 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:22.982337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 667/1351] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Sep 28 12:07:22 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:27.095460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3fc97-e7a1-4213-a5dc-6eb64a274b15 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:27.096014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3fc97-e7a1-4213-a5dc-6eb64a274b15 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:27.096014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3fc97-e7a1-4213-a5dc-6eb64a274b15 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:27.098309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 685/1352] 10.222.0.26 () {54 vars in 921 bytes} [Wed Sep 28 12:07:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:27.103871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76318d35-85bc-4f9b-a6d0-fe26ae90f212 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:27.104370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76318d35-85bc-4f9b-a6d0-fe26ae90f212 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:27.104471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76318d35-85bc-4f9b-a6d0-fe26ae90f212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:27.185455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-76318d35-85bc-4f9b-a6d0-fe26ae90f212 None None] MFA Rules not processed for user `3758ea0f26264340ad2f154c0111af21`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:27.186655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-76318d35-85bc-4f9b-a6d0-fe26ae90f212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:27.260980 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:27.299435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 668/1353] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Sep 28 12:07:27 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:28.614813 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e415c257-60f0-44d5-af13-1fda48166ed3 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:28.615349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e415c257-60f0-44d5-af13-1fda48166ed3 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:28.615349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e415c257-60f0-44d5-af13-1fda48166ed3 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:28.616714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 686/1354] 10.222.0.26 () {54 vars in 921 bytes} [Wed Sep 28 12:07:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:28.621758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b980bcfa-2d68-49e4-b918-d6f01dba6f57 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:28.622173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b980bcfa-2d68-49e4-b918-d6f01dba6f57 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:28.622312 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b980bcfa-2d68-49e4-b918-d6f01dba6f57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:28.699366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-b980bcfa-2d68-49e4-b918-d6f01dba6f57 None None] MFA Rules not processed for user `3758ea0f26264340ad2f154c0111af21`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:28.701160 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b980bcfa-2d68-49e4-b918-d6f01dba6f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:28.808291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 669/1355] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Sep 28 12:07:28 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:32.452108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833bc14a-160f-4ecd-a8bc-dce2857f9bdc None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:32.452745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833bc14a-160f-4ecd-a8bc-dce2857f9bdc None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:32.452745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833bc14a-160f-4ecd-a8bc-dce2857f9bdc None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:32.454025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 687/1356] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:32.459460 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc198c5d-cacd-4b78-a869-77093de23fb4 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:32.459871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc198c5d-cacd-4b78-a869-77093de23fb4 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:32.460108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc198c5d-cacd-4b78-a869-77093de23fb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:32.506901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:32.539430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-dc198c5d-cacd-4b78-a869-77093de23fb4 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:32.540675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-dc198c5d-cacd-4b78-a869-77093de23fb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:32.651710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 670/1357] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:32.957992 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be34c250-81b9-40c6-999e-4d2bb3a748b1 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:32.958269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be34c250-81b9-40c6-999e-4d2bb3a748b1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:32.958543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be34c250-81b9-40c6-999e-4d2bb3a748b1 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:32.960122 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 688/1358] 10.222.0.26 () {54 vars in 961 bytes} [Wed Sep 28 12:07:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:32.969789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e62cb4e-ec03-41b3-9ffb-a3adf5e12498 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:32.970380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e62cb4e-ec03-41b3-9ffb-a3adf5e12498 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:32.970946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e62cb4e-ec03-41b3-9ffb-a3adf5e12498 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:33.054356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3e62cb4e-ec03-41b3-9ffb-a3adf5e12498 None None] MFA Rules not processed for user `8bf969348e7a4158acf8fb789430b0e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:33.055538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3e62cb4e-ec03-41b3-9ffb-a3adf5e12498 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:33.105095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:33.178435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 671/1359] 10.222.0.26 () {58 vars in 1058 bytes} [Wed Sep 28 12:07:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:33.187119 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be34c250-81b9-40c6-999e-4d2bb3a748b1 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:33.188699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-be34c250-81b9-40c6-999e-4d2bb3a748b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:33.362708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be34c250-81b9-40c6-999e-4d2bb3a748b1 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:33.443028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:33.494951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df91956d-e842-4bc1-80c8-6d7745bb887e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:33.495614 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df91956d-e842-4bc1-80c8-6d7745bb887e service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:33.495844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df91956d-e842-4bc1-80c8-6d7745bb887e service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:33.496174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df91956d-e842-4bc1-80c8-6d7745bb887e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:33.497913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-df91956d-e842-4bc1-80c8-6d7745bb887e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:33.607046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df91956d-e842-4bc1-80c8-6d7745bb887e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:33.608699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df91956d-e842-4bc1-80c8-6d7745bb887e service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:33.628299 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 689/1360] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 441 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:36.029796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60155f2c-526d-48da-bf71-6ff532c2d698 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:36.029796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60155f2c-526d-48da-bf71-6ff532c2d698 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:36.030571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60155f2c-526d-48da-bf71-6ff532c2d698 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:36.031327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 672/1361] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:36.036542 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bb5547-3464-43f2-8976-229a9c1efd63 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:36.037319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bb5547-3464-43f2-8976-229a9c1efd63 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:36.037319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bb5547-3464-43f2-8976-229a9c1efd63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:36.104663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-c1bb5547-3464-43f2-8976-229a9c1efd63 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:36.105943 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c1bb5547-3464-43f2-8976-229a9c1efd63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:36.144624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:36.216346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 690/1362] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:36.508523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60155f2c-526d-48da-bf71-6ff532c2d698 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:36.510129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-60155f2c-526d-48da-bf71-6ff532c2d698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:36.678903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60155f2c-526d-48da-bf71-6ff532c2d698 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:36.690811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:36.814214 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffe1c8b-73ef-495b-8a3c-d95aeefdaaf2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:36.814214 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffe1c8b-73ef-495b-8a3c-d95aeefdaaf2 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:36.814808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffe1c8b-73ef-495b-8a3c-d95aeefdaaf2 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:36.814808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffe1c8b-73ef-495b-8a3c-d95aeefdaaf2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:36.817132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-cffe1c8b-73ef-495b-8a3c-d95aeefdaaf2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:36.914022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffe1c8b-73ef-495b-8a3c-d95aeefdaaf2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:36.915876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffe1c8b-73ef-495b-8a3c-d95aeefdaaf2 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:36.934450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 673/1363] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:39.443255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b421deff-30eb-4df9-a4ce-a9049658f34b None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:39.443838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b421deff-30eb-4df9-a4ce-a9049658f34b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:39.443838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b421deff-30eb-4df9-a4ce-a9049658f34b None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:39.445334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 691/1364] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:39.451017 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0070cf3d-f00e-48d0-9d9c-cbe6e3d0c972 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:39.451440 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0070cf3d-f00e-48d0-9d9c-cbe6e3d0c972 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:39.451618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0070cf3d-f00e-48d0-9d9c-cbe6e3d0c972 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:39.467464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:39.526817 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-0070cf3d-f00e-48d0-9d9c-cbe6e3d0c972 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:39.528593 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0070cf3d-f00e-48d0-9d9c-cbe6e3d0c972 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:39.638975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 674/1365] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:39.933234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af6f9b0-62bb-433e-b9f8-98a234628ba6 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:39.933588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af6f9b0-62bb-433e-b9f8-98a234628ba6 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:39.934482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af6f9b0-62bb-433e-b9f8-98a234628ba6 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:39.935993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 692/1366] 10.222.0.26 () {54 vars in 961 bytes} [Wed Sep 28 12:07:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:39.941372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7372d93d-832b-440f-ac57-9e111c08cafe None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:39.941666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7372d93d-832b-440f-ac57-9e111c08cafe None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:39.941945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7372d93d-832b-440f-ac57-9e111c08cafe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:40.020212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-7372d93d-832b-440f-ac57-9e111c08cafe None None] MFA Rules not processed for user `8bf969348e7a4158acf8fb789430b0e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:40.021567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-7372d93d-832b-440f-ac57-9e111c08cafe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:40.057639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:40.142072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 675/1367] 10.222.0.26 () {58 vars in 1058 bytes} [Wed Sep 28 12:07:39 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:40.149924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af6f9b0-62bb-433e-b9f8-98a234628ba6 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:40.151581 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-3af6f9b0-62bb-433e-b9f8-98a234628ba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:40.305570 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af6f9b0-62bb-433e-b9f8-98a234628ba6 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:40.326639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:40.427314 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522505f4-43a8-4867-8938-d48e4d07dd91 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:40.427924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522505f4-43a8-4867-8938-d48e4d07dd91 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:40.428195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522505f4-43a8-4867-8938-d48e4d07dd91 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:40.428488 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522505f4-43a8-4867-8938-d48e4d07dd91 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:40.430069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-522505f4-43a8-4867-8938-d48e4d07dd91 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:40.532134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522505f4-43a8-4867-8938-d48e4d07dd91 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:40.533849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522505f4-43a8-4867-8938-d48e4d07dd91 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:40.553694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 693/1368] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:40.670458 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415fb2df-ac1d-454f-8847-7cc1b8838e8d None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:40.671036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415fb2df-ac1d-454f-8847-7cc1b8838e8d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:40.671036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415fb2df-ac1d-454f-8847-7cc1b8838e8d None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:40.673254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 676/1369] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:40.680379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed22bd2-2370-451d-b9b7-4973cdcfe88b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:40.681203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed22bd2-2370-451d-b9b7-4973cdcfe88b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:40.681755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed22bd2-2370-451d-b9b7-4973cdcfe88b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:40.722739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:40.769619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-8ed22bd2-2370-451d-b9b7-4973cdcfe88b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:40.770990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8ed22bd2-2370-451d-b9b7-4973cdcfe88b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:40.898232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 694/1370] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:41.232555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c8564e-5a30-433f-9d18-9212791560f1 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:41.232845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c8564e-5a30-433f-9d18-9212791560f1 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:41.233124 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c8564e-5a30-433f-9d18-9212791560f1 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:41.235481 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 677/1371] 10.222.0.26 () {54 vars in 963 bytes} [Wed Sep 28 12:07:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:41.239750 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c22ae8-cb78-4d44-ba5a-868b949cd6c1 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:41.240143 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c22ae8-cb78-4d44-ba5a-868b949cd6c1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:41.240645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c22ae8-cb78-4d44-ba5a-868b949cd6c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:41.323903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-79c22ae8-cb78-4d44-ba5a-868b949cd6c1 None None] MFA Rules not processed for user `2ba80861b08940d398a61879bb70060a`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:41.325743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-79c22ae8-cb78-4d44-ba5a-868b949cd6c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:41.374878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:41.406046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 695/1372] 10.222.0.26 () {58 vars in 1060 bytes} [Wed Sep 28 12:07:41 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:41.413464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c8564e-5a30-433f-9d18-9212791560f1 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:41.415013 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e1c8564e-5a30-433f-9d18-9212791560f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:41.539807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c8564e-5a30-433f-9d18-9212791560f1 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:41.586383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:41.630281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26d684d-e035-4120-a498-6956cd6540c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:41.631002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d684d-e035-4120-a498-6956cd6540c5 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:41.631153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d684d-e035-4120-a498-6956cd6540c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:41.631501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d684d-e035-4120-a498-6956cd6540c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:41.633174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f26d684d-e035-4120-a498-6956cd6540c5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:41.759107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26d684d-e035-4120-a498-6956cd6540c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:41.760638 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26d684d-e035-4120-a498-6956cd6540c5 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:41.788612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 678/1373] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:42.823832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f5aeaf-0c5c-47e2-8089-3c4008df71fe None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:42.825075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f5aeaf-0c5c-47e2-8089-3c4008df71fe None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:42.825075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f5aeaf-0c5c-47e2-8089-3c4008df71fe None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:42.826406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 696/1374] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:42.832027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa2d8a-8c84-4c10-9977-cfe2a3554db3 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:42.832525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa2d8a-8c84-4c10-9977-cfe2a3554db3 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:42.832628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa2d8a-8c84-4c10-9977-cfe2a3554db3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:42.899603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-e6fa2d8a-8c84-4c10-9977-cfe2a3554db3 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:42.900852 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e6fa2d8a-8c84-4c10-9977-cfe2a3554db3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:42.921595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:43.002692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 679/1375] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:07:43.295166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f5aeaf-0c5c-47e2-8089-3c4008df71fe None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:43.296755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-90f5aeaf-0c5c-47e2-8089-3c4008df71fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:43.573290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f5aeaf-0c5c-47e2-8089-3c4008df71fe None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:43.626535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:43.700276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77646891-981e-48a8-b481-74832b92dab2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:43.700901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77646891-981e-48a8-b481-74832b92dab2 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:43.701179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77646891-981e-48a8-b481-74832b92dab2 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:43.701446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77646891-981e-48a8-b481-74832b92dab2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:43.703048 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-77646891-981e-48a8-b481-74832b92dab2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:43.800377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77646891-981e-48a8-b481-74832b92dab2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:43.801993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77646891-981e-48a8-b481-74832b92dab2 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:43.820732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 697/1376] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 526 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:46.010469 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652854a6-c907-4661-9005-ca3cf4ac097b None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:46.011029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652854a6-c907-4661-9005-ca3cf4ac097b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:46.011029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652854a6-c907-4661-9005-ca3cf4ac097b None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:46.012906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 680/1377] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:46.018193 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f3d12f-e02a-4fce-a945-429180ec8b7f None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:46.018627 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f3d12f-e02a-4fce-a945-429180ec8b7f None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:46.018735 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f3d12f-e02a-4fce-a945-429180ec8b7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:46.091091 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-d7f3d12f-e02a-4fce-a945-429180ec8b7f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:46.092562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d7f3d12f-e02a-4fce-a945-429180ec8b7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:46.099305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:46.193249 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 698/1378] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:46.493515 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652854a6-c907-4661-9005-ca3cf4ac097b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:46.495890 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-652854a6-c907-4661-9005-ca3cf4ac097b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:46.646773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652854a6-c907-4661-9005-ca3cf4ac097b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:46.650647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:46.766145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5615f47c-7e62-4fca-973b-ee01c7c82f92 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:46.766668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5615f47c-7e62-4fca-973b-ee01c7c82f92 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:46.766946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5615f47c-7e62-4fca-973b-ee01c7c82f92 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:46.767216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5615f47c-7e62-4fca-973b-ee01c7c82f92 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:46.768821 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5615f47c-7e62-4fca-973b-ee01c7c82f92 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:46.865815 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5615f47c-7e62-4fca-973b-ee01c7c82f92 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:46.867423 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5615f47c-7e62-4fca-973b-ee01c7c82f92 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:46.886391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 681/1379] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:49.162317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96712d5-ecd6-49bb-b0c4-c317f4d450df None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:49.162962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96712d5-ecd6-49bb-b0c4-c317f4d450df None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:49.162962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96712d5-ecd6-49bb-b0c4-c317f4d450df None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:49.170720 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d5a238-7f84-402f-aa2b-79e824f23d38 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:49.171386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d5a238-7f84-402f-aa2b-79e824f23d38 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:49.171386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d5a238-7f84-402f-aa2b-79e824f23d38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:49.176563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 699/1380] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:49 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:49.183581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:49.244042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-10d5a238-7f84-402f-aa2b-79e824f23d38 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:49.245362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-10d5a238-7f84-402f-aa2b-79e824f23d38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:49.351031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 682/1381] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:07:49.720932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96712d5-ecd6-49bb-b0c4-c317f4d450df None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:49.722442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f96712d5-ecd6-49bb-b0c4-c317f4d450df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:49.874514 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:49.888425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96712d5-ecd6-49bb-b0c4-c317f4d450df None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:50.006359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f50d49-2a60-4610-800f-93ebc33f343c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:50.006890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f50d49-2a60-4610-800f-93ebc33f343c service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:50.007171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f50d49-2a60-4610-800f-93ebc33f343c service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:50.007458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f50d49-2a60-4610-800f-93ebc33f343c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:50.009118 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-90f50d49-2a60-4610-800f-93ebc33f343c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:50.104123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f50d49-2a60-4610-800f-93ebc33f343c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:50.106283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f50d49-2a60-4610-800f-93ebc33f343c service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:50.116137 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:50.125224 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 700/1382] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:52.337347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea5b2ba-f226-413e-b2ea-bb0c19bb2d65 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:52.338050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea5b2ba-f226-413e-b2ea-bb0c19bb2d65 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:52.338050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea5b2ba-f226-413e-b2ea-bb0c19bb2d65 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:52.339241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 683/1383] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:52.345330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9dab1e-4031-44ab-9c95-0e1162a858bb None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:52.345751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9dab1e-4031-44ab-9c95-0e1162a858bb None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:52.345863 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9dab1e-4031-44ab-9c95-0e1162a858bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:52.410105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-ac9dab1e-4031-44ab-9c95-0e1162a858bb None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:52.411299 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ac9dab1e-4031-44ab-9c95-0e1162a858bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:52.510917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 701/1384] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:52.796277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea5b2ba-f226-413e-b2ea-bb0c19bb2d65 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:52.797814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1ea5b2ba-f226-413e-b2ea-bb0c19bb2d65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:52.885794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:52.947017 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea5b2ba-f226-413e-b2ea-bb0c19bb2d65 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:53.060585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0493da13-def0-4a4c-8092-a1e110731bbe service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:53.061110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0493da13-def0-4a4c-8092-a1e110731bbe service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:53.061438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0493da13-def0-4a4c-8092-a1e110731bbe service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:53.061710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0493da13-def0-4a4c-8092-a1e110731bbe service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:53.063284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0493da13-def0-4a4c-8092-a1e110731bbe service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:53.113406 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:53.158152 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0493da13-def0-4a4c-8092-a1e110731bbe service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:53.160255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0493da13-def0-4a4c-8092-a1e110731bbe service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:53.180374 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 684/1385] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:55.338017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec42b0b-2ab8-492e-9940-92b2058cc693 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:55.338578 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec42b0b-2ab8-492e-9940-92b2058cc693 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:55.338578 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec42b0b-2ab8-492e-9940-92b2058cc693 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:55.340396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 702/1386] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:55.345934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb28977-6e13-4b65-8f59-2af19fda3efb None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:55.346336 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb28977-6e13-4b65-8f59-2af19fda3efb None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:55.346530 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb28977-6e13-4b65-8f59-2af19fda3efb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:55.411036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-bbb28977-6e13-4b65-8f59-2af19fda3efb None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:55.412228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-bbb28977-6e13-4b65-8f59-2af19fda3efb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:55.508890 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 685/1387] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:55.794039 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec42b0b-2ab8-492e-9940-92b2058cc693 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:55.795520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0ec42b0b-2ab8-492e-9940-92b2058cc693 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:55.873112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:55.941598 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec42b0b-2ab8-492e-9940-92b2058cc693 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:56.054018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51063740-4d01-486b-b05b-d2c7fe5e2539 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:56.054537 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51063740-4d01-486b-b05b-d2c7fe5e2539 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:56.054810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51063740-4d01-486b-b05b-d2c7fe5e2539 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:56.055076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51063740-4d01-486b-b05b-d2c7fe5e2539 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:56.056671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-51063740-4d01-486b-b05b-d2c7fe5e2539 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:56.095556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:56.146350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51063740-4d01-486b-b05b-d2c7fe5e2539 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:56.147916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51063740-4d01-486b-b05b-d2c7fe5e2539 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:56.165849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 703/1388] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:58.348576 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d082de-efa7-4506-a53d-85322d83bae7 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:58.349120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d082de-efa7-4506-a53d-85322d83bae7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:58.349120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d082de-efa7-4506-a53d-85322d83bae7 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:58.350454 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 686/1389] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:58.355671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c88f8e-69a3-46bc-aa26-ded586cbf3a6 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:58.356121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c88f8e-69a3-46bc-aa26-ded586cbf3a6 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:58.356254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c88f8e-69a3-46bc-aa26-ded586cbf3a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:58.425307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-87c88f8e-69a3-46bc-aa26-ded586cbf3a6 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:58.426522 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-87c88f8e-69a3-46bc-aa26-ded586cbf3a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:58.540753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 704/1390] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:07:58.845996 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d082de-efa7-4506-a53d-85322d83bae7 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:07:58.847524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b9d082de-efa7-4506-a53d-85322d83bae7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:58.883530 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:58.999924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d082de-efa7-4506-a53d-85322d83bae7 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:07:59.100210 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:59.112819 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76794f9-4746-430a-9e39-3e1c876b40fe service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:07:59.113429 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76794f9-4746-430a-9e39-3e1c876b40fe service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:59.113711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76794f9-4746-430a-9e39-3e1c876b40fe service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:59.113985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76794f9-4746-430a-9e39-3e1c876b40fe service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:59.115574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c76794f9-4746-430a-9e39-3e1c876b40fe service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:59.213032 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76794f9-4746-430a-9e39-3e1c876b40fe service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:07:59.214645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76794f9-4746-430a-9e39-3e1c876b40fe service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:07:59.233589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 687/1391] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:07:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:07:59.694811 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa41edd-67cb-4dd4-b561-367eef111e2b None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:59.695350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa41edd-67cb-4dd4-b561-367eef111e2b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:59.695350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa41edd-67cb-4dd4-b561-367eef111e2b None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:59.696697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 705/1392] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:07:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:07:59.702590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edddaa1-7a6a-4ccf-a02e-ac6a52efcc14 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:07:59.703064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edddaa1-7a6a-4ccf-a02e-ac6a52efcc14 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:07:59.703175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edddaa1-7a6a-4ccf-a02e-ac6a52efcc14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:07:59.770239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-4edddaa1-7a6a-4ccf-a02e-ac6a52efcc14 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:07:59.771444 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-4edddaa1-7a6a-4ccf-a02e-ac6a52efcc14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:07:59.831634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:07:59.869277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 688/1393] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:07:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:00.171063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1346df30-c0d1-4d9a-bad2-bc13c6435cf3 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:00.171338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1346df30-c0d1-4d9a-bad2-bc13c6435cf3 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:00.171611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1346df30-c0d1-4d9a-bad2-bc13c6435cf3 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:00.172965 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 706/1394] 10.222.0.26 () {54 vars in 962 bytes} [Wed Sep 28 12:08:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:00.178136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb938f1-266a-40dd-ae18-7f67e4f521f0 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:00.178496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb938f1-266a-40dd-ae18-7f67e4f521f0 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:00.178842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb938f1-266a-40dd-ae18-7f67e4f521f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:00.253540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-dcb938f1-266a-40dd-ae18-7f67e4f521f0 None None] MFA Rules not processed for user `815504693bcc40b6aef06aacb2a9a35b`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:00.254747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-dcb938f1-266a-40dd-ae18-7f67e4f521f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:00.328217 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 689/1395] 10.222.0.26 () {58 vars in 1059 bytes} [Wed Sep 28 12:08:00 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:00.337401 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1346df30-c0d1-4d9a-bad2-bc13c6435cf3 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:00.339071 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1346df30-c0d1-4d9a-bad2-bc13c6435cf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:00.352841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:00.441696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1346df30-c0d1-4d9a-bad2-bc13c6435cf3 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:00.536365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e751b9d3-c6d2-4887-9d78-74fabf2090c6 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '815504693bcc40b6aef06aacb2a9a35b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:00.537294 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e751b9d3-c6d2-4887-9d78-74fabf2090c6 service glance] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:00.538887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e751b9d3-c6d2-4887-9d78-74fabf2090c6 service glance] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:00.538887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e751b9d3-c6d2-4887-9d78-74fabf2090c6 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:00.539588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e751b9d3-c6d2-4887-9d78-74fabf2090c6 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:00.617530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:00.663963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e751b9d3-c6d2-4887-9d78-74fabf2090c6 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:00.665608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e751b9d3-c6d2-4887-9d78-74fabf2090c6 service glance] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:00.685757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 707/1396] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:01.454106 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc5d23f-5a7c-4546-83c4-3e2e88fc5a0a None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:01.454937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc5d23f-5a7c-4546-83c4-3e2e88fc5a0a None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:01.455414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc5d23f-5a7c-4546-83c4-3e2e88fc5a0a None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:01.460349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 690/1397] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:08:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:01.461827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7fc553-4c8c-4e6b-b11a-6903a46acf2c None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:01.462115 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7fc553-4c8c-4e6b-b11a-6903a46acf2c None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:01.462241 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d405f20-5152-4d6a-83e1-ee7f33cffa0d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:01.462640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7fc553-4c8c-4e6b-b11a-6903a46acf2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:01.462743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d405f20-5152-4d6a-83e1-ee7f33cffa0d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:01.462743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d405f20-5152-4d6a-83e1-ee7f33cffa0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:01.535511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-8e7fc553-4c8c-4e6b-b11a-6903a46acf2c None None] MFA Rules not processed for user `b20a6fb4e7a546058f5fa314c9a59299`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:01.536770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8e7fc553-4c8c-4e6b-b11a-6903a46acf2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:01.537081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-6d405f20-5152-4d6a-83e1-ee7f33cffa0d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:01.538418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6d405f20-5152-4d6a-83e1-ee7f33cffa0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:01.546502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:01.628176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 691/1398] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Sep 28 12:08:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:08:01.641847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 708/1399] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:08:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:01.673807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93e45cb-c85b-40f9-b08b-b03010307047 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:01.674096 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93e45cb-c85b-40f9-b08b-b03010307047 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:01.674368 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93e45cb-c85b-40f9-b08b-b03010307047 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:01.675743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 692/1400] 10.222.0.26 () {54 vars in 943 bytes} [Wed Sep 28 12:08:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:01.682466 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00b0c1f-7d0b-4bdb-86c7-3bafb2f4c010 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:01.682749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00b0c1f-7d0b-4bdb-86c7-3bafb2f4c010 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:01.683017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00b0c1f-7d0b-4bdb-86c7-3bafb2f4c010 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:01.709955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:01.762183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-c00b0c1f-7d0b-4bdb-86c7-3bafb2f4c010 None None] MFA Rules not processed for user `e663aad79bb8429aa223564a15332ee2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:01.763515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c00b0c1f-7d0b-4bdb-86c7-3bafb2f4c010 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:01.877311 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 709/1401] 10.222.0.26 () {58 vars in 1040 bytes} [Wed Sep 28 12:08:01 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:01.889222 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93e45cb-c85b-40f9-b08b-b03010307047 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:01.890867 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d93e45cb-c85b-40f9-b08b-b03010307047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:01.933912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00b0c1f-7d0b-4bdb-86c7-3bafb2f4c010 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:01.935149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c00b0c1f-7d0b-4bdb-86c7-3bafb2f4c010 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:02.012459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:02.031648 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93e45cb-c85b-40f9-b08b-b03010307047 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:02.035924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00b0c1f-7d0b-4bdb-86c7-3bafb2f4c010 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:02.048871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:02.139317 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0b725f-e27a-40fe-a38a-e19374109126 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e663aad79bb8429aa223564a15332ee2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:02.139930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0b725f-e27a-40fe-a38a-e19374109126 service swift] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:02.140209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0b725f-e27a-40fe-a38a-e19374109126 service swift] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:02.140510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0b725f-e27a-40fe-a38a-e19374109126 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:02.142142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2f0b725f-e27a-40fe-a38a-e19374109126 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:02.149584 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf5cf1c-0c69-4010-9f00-084026c24d40 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:02.150297 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf5cf1c-0c69-4010-9f00-084026c24d40 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:02.150648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf5cf1c-0c69-4010-9f00-084026c24d40 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:02.151160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf5cf1c-0c69-4010-9f00-084026c24d40 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:02.153056 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-acf5cf1c-0c69-4010-9f00-084026c24d40 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:02.227861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0b725f-e27a-40fe-a38a-e19374109126 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:02.229501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0b725f-e27a-40fe-a38a-e19374109126 service swift] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:02.238773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 693/1402] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Sep 28 12:08:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 28 12:08:02.248165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf5cf1c-0c69-4010-9f00-084026c24d40 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:02.249770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf5cf1c-0c69-4010-9f00-084026c24d40 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:02.254927 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:02.269042 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 710/1403] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:08:04.597682 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483f350b-d7e4-4241-908a-7217a25a339b None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:04.598233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483f350b-d7e4-4241-908a-7217a25a339b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:04.598233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483f350b-d7e4-4241-908a-7217a25a339b None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:04.599612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 694/1404] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:08:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:04.605164 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72458c2-ca97-472f-bf89-088016195536 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:04.606744 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72458c2-ca97-472f-bf89-088016195536 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:04.607254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72458c2-ca97-472f-bf89-088016195536 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:04.678407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-c72458c2-ca97-472f-bf89-088016195536 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:04.679606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c72458c2-ca97-472f-bf89-088016195536 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:04.786134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 711/1405] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:08:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:05.076982 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483f350b-d7e4-4241-908a-7217a25a339b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:05.078491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-483f350b-d7e4-4241-908a-7217a25a339b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:05.138580 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:05.246861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483f350b-d7e4-4241-908a-7217a25a339b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:05.368707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45536052-c7c6-42bd-858e-14fe7a61e2d8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:05.369316 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45536052-c7c6-42bd-858e-14fe7a61e2d8 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:05.369601 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45536052-c7c6-42bd-858e-14fe7a61e2d8 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:05.369876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45536052-c7c6-42bd-858e-14fe7a61e2d8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:05.371503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-45536052-c7c6-42bd-858e-14fe7a61e2d8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:05.384593 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:05.473792 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45536052-c7c6-42bd-858e-14fe7a61e2d8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:05.475637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45536052-c7c6-42bd-858e-14fe7a61e2d8 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:05.506446 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 695/1406] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:07.711233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b8c308-c3be-411f-8f63-c992e31319bc None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:07.711767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b8c308-c3be-411f-8f63-c992e31319bc None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:07.711767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b8c308-c3be-411f-8f63-c992e31319bc None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:07.716676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 712/1407] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:08:07 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:07.719353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b006d1e8-72e3-4f05-9657-d08346613f18 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:07.719788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b006d1e8-72e3-4f05-9657-d08346613f18 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:07.719962 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b006d1e8-72e3-4f05-9657-d08346613f18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:07.791936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-b006d1e8-72e3-4f05-9657-d08346613f18 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:07.793835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b006d1e8-72e3-4f05-9657-d08346613f18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:07.881714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:07.901475 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 696/1408] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:08:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:08.196128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b8c308-c3be-411f-8f63-c992e31319bc None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:08.197771 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d3b8c308-c3be-411f-8f63-c992e31319bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:08.278855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:08.359873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b8c308-c3be-411f-8f63-c992e31319bc None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:08.492019 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d400b90-7aa7-4ec5-b132-9474fcd9b5e4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:08.492573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d400b90-7aa7-4ec5-b132-9474fcd9b5e4 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:08.492847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d400b90-7aa7-4ec5-b132-9474fcd9b5e4 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:08.493107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d400b90-7aa7-4ec5-b132-9474fcd9b5e4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:08.496921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-9d400b90-7aa7-4ec5-b132-9474fcd9b5e4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:08.534287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:08.599043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d400b90-7aa7-4ec5-b132-9474fcd9b5e4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:08.601632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d400b90-7aa7-4ec5-b132-9474fcd9b5e4 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:08.622512 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 713/1409] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:17.899985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=6 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:18.619281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=31 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:25.272779 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08abc4e9-8c57-4a7c-a163-7d2b64689a55 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:25.273388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08abc4e9-8c57-4a7c-a163-7d2b64689a55 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:25.273388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08abc4e9-8c57-4a7c-a163-7d2b64689a55 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:25.274688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 697/1410] 10.222.0.26 () {54 vars in 918 bytes} [Wed Sep 28 12:08:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:25.280267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057ab2d-dc11-4839-96ce-acc3685c826d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:25.280727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057ab2d-dc11-4839-96ce-acc3685c826d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:25.280829 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057ab2d-dc11-4839-96ce-acc3685c826d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:25.354020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-6057ab2d-dc11-4839-96ce-acc3685c826d None None] MFA Rules not processed for user `3758ea0f26264340ad2f154c0111af21`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:25.355844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6057ab2d-dc11-4839-96ce-acc3685c826d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:25.458259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 714/1411] 10.222.0.26 () {58 vars in 1015 bytes} [Wed Sep 28 12:08:25 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:25.473282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c645ae-ae04-4d61-8b0f-3f4ec11676ba None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:25.473573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c645ae-ae04-4d61-8b0f-3f4ec11676ba None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:25.473868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c645ae-ae04-4d61-8b0f-3f4ec11676ba None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:25.475212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 698/1412] 10.222.0.26 () {54 vars in 918 bytes} [Wed Sep 28 12:08:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:25.480334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb08fe7-9c96-4a2f-b996-43231405149a None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:25.480633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb08fe7-9c96-4a2f-b996-43231405149a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:25.480904 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb08fe7-9c96-4a2f-b996-43231405149a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:25.557530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:25.560273 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-1eb08fe7-9c96-4a2f-b996-43231405149a None None] MFA Rules not processed for user `8bf969348e7a4158acf8fb789430b0e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:25.562135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1eb08fe7-9c96-4a2f-b996-43231405149a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:25.674357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 715/1413] 10.222.0.26 () {58 vars in 1015 bytes} [Wed Sep 28 12:08:25 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:28.222392 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9b8cd6-9afc-43d5-a7a0-641914c0ce58 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:28.222972 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9b8cd6-9afc-43d5-a7a0-641914c0ce58 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:28.222972 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9b8cd6-9afc-43d5-a7a0-641914c0ce58 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:28.224280 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 699/1414] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:08:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:28.230171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8b1c9d-4e38-409b-908c-af41b31d2c39 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:28.230552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8b1c9d-4e38-409b-908c-af41b31d2c39 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:28.230722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8b1c9d-4e38-409b-908c-af41b31d2c39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:28.296531 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-ca8b1c9d-4e38-409b-908c-af41b31d2c39 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:28.297721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ca8b1c9d-4e38-409b-908c-af41b31d2c39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:28.351235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:28.391084 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 716/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:08:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:28.681562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9b8cd6-9afc-43d5-a7a0-641914c0ce58 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:28.683720 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0f9b8cd6-9afc-43d5-a7a0-641914c0ce58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:28.828908 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9b8cd6-9afc-43d5-a7a0-641914c0ce58 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:28.901942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:28.941553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e632fa-06a7-4299-ba4b-9d3c59bc5338 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:28.942155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e632fa-06a7-4299-ba4b-9d3c59bc5338 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:28.942432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e632fa-06a7-4299-ba4b-9d3c59bc5338 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:28.942702 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e632fa-06a7-4299-ba4b-9d3c59bc5338 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:28.944275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a4e632fa-06a7-4299-ba4b-9d3c59bc5338 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:29.031514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e632fa-06a7-4299-ba4b-9d3c59bc5338 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:29.033134 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e632fa-06a7-4299-ba4b-9d3c59bc5338 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:29.050492 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 700/1416] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:30.206234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb62164e-ee93-4f40-aded-b7ac05ea1c20 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:30.206805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb62164e-ee93-4f40-aded-b7ac05ea1c20 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:30.206805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb62164e-ee93-4f40-aded-b7ac05ea1c20 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:30.208108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 717/1417] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 28 12:08:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:30.214414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84418892-3b0c-44d9-a9ef-dc64594fbeee None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:30.214854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84418892-3b0c-44d9-a9ef-dc64594fbeee None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:30.214972 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84418892-3b0c-44d9-a9ef-dc64594fbeee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:30.288095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-84418892-3b0c-44d9-a9ef-dc64594fbeee None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:30.289323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-84418892-3b0c-44d9-a9ef-dc64594fbeee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:30.316114 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:30.384294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 701/1418] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 28 12:08:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:30.551808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb62164e-ee93-4f40-aded-b7ac05ea1c20 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:30.553735 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-cb62164e-ee93-4f40-aded-b7ac05ea1c20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:30.702912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb62164e-ee93-4f40-aded-b7ac05ea1c20 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:30.743231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:30.818887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c1e176-2e65-41c4-99ae-cc4e870eb8ca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:30.819496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c1e176-2e65-41c4-99ae-cc4e870eb8ca service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:30.819772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c1e176-2e65-41c4-99ae-cc4e870eb8ca service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:30.820044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c1e176-2e65-41c4-99ae-cc4e870eb8ca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:30.821716 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-61c1e176-2e65-41c4-99ae-cc4e870eb8ca service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:30.910486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c1e176-2e65-41c4-99ae-cc4e870eb8ca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:30.912062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c1e176-2e65-41c4-99ae-cc4e870eb8ca service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:30.930349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 718/1419] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:32.062869 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d5fe6d-6b0f-4051-ac18-9d3eb02ad628 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:32.063435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d5fe6d-6b0f-4051-ac18-9d3eb02ad628 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:32.063435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d5fe6d-6b0f-4051-ac18-9d3eb02ad628 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:32.064791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 702/1420] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 28 12:08:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:32.070993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05897991-f412-42a2-b9e2-eb9b12fac2cd None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:32.071429 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05897991-f412-42a2-b9e2-eb9b12fac2cd None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:32.071544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05897991-f412-42a2-b9e2-eb9b12fac2cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:32.124891 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:32.148757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-05897991-f412-42a2-b9e2-eb9b12fac2cd None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:32.150609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-05897991-f412-42a2-b9e2-eb9b12fac2cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:32.251052 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 719/1421] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 28 12:08:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:08:32.412742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d5fe6d-6b0f-4051-ac18-9d3eb02ad628 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:32.414281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-74d5fe6d-6b0f-4051-ac18-9d3eb02ad628 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:32.563615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d5fe6d-6b0f-4051-ac18-9d3eb02ad628 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:32.576145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:32.679812 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c02575e-3dcc-4ab6-9b01-496f59093f7c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:32.680344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c02575e-3dcc-4ab6-9b01-496f59093f7c service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:32.680628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c02575e-3dcc-4ab6-9b01-496f59093f7c service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:32.680898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c02575e-3dcc-4ab6-9b01-496f59093f7c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:32.683100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3c02575e-3dcc-4ab6-9b01-496f59093f7c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:32.770678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c02575e-3dcc-4ab6-9b01-496f59093f7c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:32.772272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c02575e-3dcc-4ab6-9b01-496f59093f7c service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:32.790243 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 703/1422] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:33.905510 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa9117e-74d8-4b13-9f2f-db1df533f87f None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:33.906077 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa9117e-74d8-4b13-9f2f-db1df533f87f None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:33.906077 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa9117e-74d8-4b13-9f2f-db1df533f87f None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:33.907395 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 720/1423] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 28 12:08:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:33.913591 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21890d9-03e6-4e5e-9195-4160c849ab7e None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:33.914002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21890d9-03e6-4e5e-9195-4160c849ab7e None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:33.914112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21890d9-03e6-4e5e-9195-4160c849ab7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:33.934372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:33.991765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-b21890d9-03e6-4e5e-9195-4160c849ab7e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:33.993021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b21890d9-03e6-4e5e-9195-4160c849ab7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:34.091686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 704/1424] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 28 12:08:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:34.243199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa9117e-74d8-4b13-9f2f-db1df533f87f None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:34.244764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4aa9117e-74d8-4b13-9f2f-db1df533f87f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:34.358214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:34.393152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa9117e-74d8-4b13-9f2f-db1df533f87f None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:34.508905 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cd4165-4bca-4591-b213-bb12b4413697 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:34.509643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cd4165-4bca-4591-b213-bb12b4413697 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:34.509919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cd4165-4bca-4591-b213-bb12b4413697 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:34.510189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cd4165-4bca-4591-b213-bb12b4413697 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:34.512103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-17cd4165-4bca-4591-b213-bb12b4413697 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:34.591254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:34.606020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cd4165-4bca-4591-b213-bb12b4413697 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:34.607630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cd4165-4bca-4591-b213-bb12b4413697 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:34.625056 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 721/1425] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:35.763555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b32064-ca95-4e67-ab6c-6cb5883475eb None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:35.764099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b32064-ca95-4e67-ab6c-6cb5883475eb None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:35.764223 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b32064-ca95-4e67-ab6c-6cb5883475eb None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:35.765540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 705/1426] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 28 12:08:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:35.771882 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be9e8e4-df15-43a5-9e2e-6d978c18e187 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:35.772387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be9e8e4-df15-43a5-9e2e-6d978c18e187 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:35.772505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be9e8e4-df15-43a5-9e2e-6d978c18e187 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:35.852592 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-6be9e8e4-df15-43a5-9e2e-6d978c18e187 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:35.853853 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6be9e8e4-df15-43a5-9e2e-6d978c18e187 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:35.948797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 722/1427] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 28 12:08:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:36.104770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b32064-ca95-4e67-ab6c-6cb5883475eb None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:36.106331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-90b32064-ca95-4e67-ab6c-6cb5883475eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:36.190529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:36.251023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b32064-ca95-4e67-ab6c-6cb5883475eb None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:36.360174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed80d8d-8229-44df-989d-eba78b88df39 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:36.360794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed80d8d-8229-44df-989d-eba78b88df39 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:36.361074 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed80d8d-8229-44df-989d-eba78b88df39 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:36.361345 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed80d8d-8229-44df-989d-eba78b88df39 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:36.362917 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0ed80d8d-8229-44df-989d-eba78b88df39 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:36.410345 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:36.450382 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed80d8d-8229-44df-989d-eba78b88df39 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:36.451968 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed80d8d-8229-44df-989d-eba78b88df39 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:36.470289 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 706/1428] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:37.524207 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276eb786-f120-4b24-924a-292e38ff0da0 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:37.524719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276eb786-f120-4b24-924a-292e38ff0da0 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:37.524824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276eb786-f120-4b24-924a-292e38ff0da0 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:37.526144 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 723/1429] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 28 12:08:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:37.532433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bff59d-e703-4b10-9018-0178aad5e324 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:37.532869 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bff59d-e703-4b10-9018-0178aad5e324 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:37.532986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bff59d-e703-4b10-9018-0178aad5e324 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:37.608252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-21bff59d-e703-4b10-9018-0178aad5e324 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:37.609472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-21bff59d-e703-4b10-9018-0178aad5e324 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:37.703878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 707/1430] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 28 12:08:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:37.859642 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276eb786-f120-4b24-924a-292e38ff0da0 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:37.861256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-276eb786-f120-4b24-924a-292e38ff0da0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:37.908977 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:38.009032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276eb786-f120-4b24-924a-292e38ff0da0 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:38.132004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18791065-a752-4330-8657-d73dd704f8d3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:38.132634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18791065-a752-4330-8657-d73dd704f8d3 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:38.132912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18791065-a752-4330-8657-d73dd704f8d3 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:38.133178 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18791065-a752-4330-8657-d73dd704f8d3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:38.134758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-18791065-a752-4330-8657-d73dd704f8d3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:38.141633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:38.222682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18791065-a752-4330-8657-d73dd704f8d3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:38.224275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18791065-a752-4330-8657-d73dd704f8d3 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:38.241993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 724/1431] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:39.302729 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca3c936-03cd-475a-a45f-aafb446cdbf2 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:39.303318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca3c936-03cd-475a-a45f-aafb446cdbf2 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:39.303318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca3c936-03cd-475a-a45f-aafb446cdbf2 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:39.304655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 708/1432] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 28 12:08:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:39.311552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b48379-8664-4371-bf00-67f9fd5b164b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:39.311976 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b48379-8664-4371-bf00-67f9fd5b164b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:39.312087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b48379-8664-4371-bf00-67f9fd5b164b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:39.386088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-75b48379-8664-4371-bf00-67f9fd5b164b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:39.387280 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-75b48379-8664-4371-bf00-67f9fd5b164b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:39.453532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:39.482925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 725/1433] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 28 12:08:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:39.633389 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca3c936-03cd-475a-a45f-aafb446cdbf2 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:39.634829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-4ca3c936-03cd-475a-a45f-aafb446cdbf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:39.652708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:39.781009 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca3c936-03cd-475a-a45f-aafb446cdbf2 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:39.865847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:39.892249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f2a743-2aaf-468c-b428-cfd9b8c1d7b2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:39.892818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f2a743-2aaf-468c-b428-cfd9b8c1d7b2 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:39.893097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f2a743-2aaf-468c-b428-cfd9b8c1d7b2 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:39.893375 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f2a743-2aaf-468c-b428-cfd9b8c1d7b2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:39.894958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-13f2a743-2aaf-468c-b428-cfd9b8c1d7b2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:39.982333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f2a743-2aaf-468c-b428-cfd9b8c1d7b2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:39.983931 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f2a743-2aaf-468c-b428-cfd9b8c1d7b2 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:40.002167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 709/1434] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:41.062504 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65adee95-bd7d-4aab-b398-9c92aab54dae None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:41.063099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65adee95-bd7d-4aab-b398-9c92aab54dae None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:41.063099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65adee95-bd7d-4aab-b398-9c92aab54dae None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:41.064425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 726/1435] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 28 12:08:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:41.070434 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6dcb92-66d4-4cbd-8ffd-3d5975460d46 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:41.070876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6dcb92-66d4-4cbd-8ffd-3d5975460d46 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:41.070989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6dcb92-66d4-4cbd-8ffd-3d5975460d46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:41.145456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-4d6dcb92-66d4-4cbd-8ffd-3d5975460d46 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:41.146656 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-4d6dcb92-66d4-4cbd-8ffd-3d5975460d46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:41.189007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:41.243371 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 710/1436] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 28 12:08:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:41.396135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65adee95-bd7d-4aab-b398-9c92aab54dae None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:41.397658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-65adee95-bd7d-4aab-b398-9c92aab54dae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:41.539775 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65adee95-bd7d-4aab-b398-9c92aab54dae None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:41.592449 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:41.649795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ccadc6-0877-4427-ba36-7b2fcbebc372 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:41.650359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ccadc6-0877-4427-ba36-7b2fcbebc372 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:41.650656 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ccadc6-0877-4427-ba36-7b2fcbebc372 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:41.650953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ccadc6-0877-4427-ba36-7b2fcbebc372 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:41.652556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-58ccadc6-0877-4427-ba36-7b2fcbebc372 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:41.739944 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ccadc6-0877-4427-ba36-7b2fcbebc372 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:41.741568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ccadc6-0877-4427-ba36-7b2fcbebc372 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:41.759406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 727/1437] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:42.803004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb5baf1-cf45-4a25-ade2-0b0183e43b68 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:42.803531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb5baf1-cf45-4a25-ade2-0b0183e43b68 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:42.803628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb5baf1-cf45-4a25-ade2-0b0183e43b68 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:42.804940 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 711/1438] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 28 12:08:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:42.811231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e765f46c-1976-4252-a6cd-7f39dfab858d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:42.811669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e765f46c-1976-4252-a6cd-7f39dfab858d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:42.811785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e765f46c-1976-4252-a6cd-7f39dfab858d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:42.887703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-e765f46c-1976-4252-a6cd-7f39dfab858d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:42.888935 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e765f46c-1976-4252-a6cd-7f39dfab858d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:42.890895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:42.983240 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 728/1439] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 28 12:08:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:43.133053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb5baf1-cf45-4a25-ade2-0b0183e43b68 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:43.134501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0fb5baf1-cf45-4a25-ade2-0b0183e43b68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:43.281233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb5baf1-cf45-4a25-ade2-0b0183e43b68 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:43.307519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:43.391192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e02a08-9250-4854-928d-9d4fc81f6688 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:43.391737 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e02a08-9250-4854-928d-9d4fc81f6688 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:43.392015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e02a08-9250-4854-928d-9d4fc81f6688 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:43.392283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e02a08-9250-4854-928d-9d4fc81f6688 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:43.393871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-49e02a08-9250-4854-928d-9d4fc81f6688 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:43.481061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e02a08-9250-4854-928d-9d4fc81f6688 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:43.482651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e02a08-9250-4854-928d-9d4fc81f6688 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:43.500799 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 712/1440] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:08:45.595636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b243e19e-400a-4a46-8582-f4cf0ba20231 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:45.596333 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b243e19e-400a-4a46-8582-f4cf0ba20231 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:45.596333 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b243e19e-400a-4a46-8582-f4cf0ba20231 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:45.597590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 729/1441] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:08:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:45.603226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df89eab-43ba-4ef6-a049-914374defdc2 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:45.604352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df89eab-43ba-4ef6-a049-914374defdc2 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:45.604460 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df89eab-43ba-4ef6-a049-914374defdc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:45.648880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:45.674579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-1df89eab-43ba-4ef6-a049-914374defdc2 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:45.675810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1df89eab-43ba-4ef6-a049-914374defdc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:45.770169 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 713/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:08:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:46.062902 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b243e19e-400a-4a46-8582-f4cf0ba20231 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:46.064450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b243e19e-400a-4a46-8582-f4cf0ba20231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:46.170405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b243e19e-400a-4a46-8582-f4cf0ba20231 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:46.202341 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:46.260728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43744a2e-675b-44da-9b6e-67b965480cad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:46.269325 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43744a2e-675b-44da-9b6e-67b965480cad service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:46.269604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43744a2e-675b-44da-9b6e-67b965480cad service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:46.269888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43744a2e-675b-44da-9b6e-67b965480cad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:46.271636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-43744a2e-675b-44da-9b6e-67b965480cad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:46.385622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43744a2e-675b-44da-9b6e-67b965480cad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:46.387057 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43744a2e-675b-44da-9b6e-67b965480cad service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:46.404594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 730/1443] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:48.682546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8f6cd5-0ac5-4e38-bab6-af34c49ffa29 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:48.683113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8f6cd5-0ac5-4e38-bab6-af34c49ffa29 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:48.683113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8f6cd5-0ac5-4e38-bab6-af34c49ffa29 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:48.684449 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 714/1444] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:08:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:48.689610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7873b215-2b30-4134-a39a-b8c355f23dbe None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:48.690056 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7873b215-2b30-4134-a39a-b8c355f23dbe None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:48.690168 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7873b215-2b30-4134-a39a-b8c355f23dbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:48.754230 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-7873b215-2b30-4134-a39a-b8c355f23dbe None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:48.755334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7873b215-2b30-4134-a39a-b8c355f23dbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:48.768260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:48.850010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 731/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:08:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:49.130456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8f6cd5-0ac5-4e38-bab6-af34c49ffa29 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:49.132007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-cd8f6cd5-0ac5-4e38-bab6-af34c49ffa29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:49.234151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8f6cd5-0ac5-4e38-bab6-af34c49ffa29 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:49.247047 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:49.307054 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a827b172-4c4b-40b6-9f3f-420c70d81981 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:49.307658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a827b172-4c4b-40b6-9f3f-420c70d81981 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:49.307936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a827b172-4c4b-40b6-9f3f-420c70d81981 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:49.308206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a827b172-4c4b-40b6-9f3f-420c70d81981 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:49.309806 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a827b172-4c4b-40b6-9f3f-420c70d81981 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:49.415897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a827b172-4c4b-40b6-9f3f-420c70d81981 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:49.417381 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a827b172-4c4b-40b6-9f3f-420c70d81981 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:49.435207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 715/1446] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:51.906218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cedcea2-8101-4ef6-ac23-94b1b2d5cb06 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:51.906785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cedcea2-8101-4ef6-ac23-94b1b2d5cb06 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:51.906785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cedcea2-8101-4ef6-ac23-94b1b2d5cb06 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:51.908195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 732/1447] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:08:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:51.913428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c6285-1dfa-4350-b746-317dbc9a1d74 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:51.913871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c6285-1dfa-4350-b746-317dbc9a1d74 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:51.913977 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c6285-1dfa-4350-b746-317dbc9a1d74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:51.961549 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:51.980176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-d65c6285-1dfa-4350-b746-317dbc9a1d74 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:51.981411 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d65c6285-1dfa-4350-b746-317dbc9a1d74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:52.077182 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 716/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:08:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:52.360901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cedcea2-8101-4ef6-ac23-94b1b2d5cb06 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:52.362414 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6cedcea2-8101-4ef6-ac23-94b1b2d5cb06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:52.462148 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cedcea2-8101-4ef6-ac23-94b1b2d5cb06 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:52.502599 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:52.536541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe572b39-0b3c-4e01-bb71-60fceaac3fe0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:52.537140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe572b39-0b3c-4e01-bb71-60fceaac3fe0 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:52.537419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe572b39-0b3c-4e01-bb71-60fceaac3fe0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:52.537686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe572b39-0b3c-4e01-bb71-60fceaac3fe0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:52.539256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-fe572b39-0b3c-4e01-bb71-60fceaac3fe0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:52.654956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe572b39-0b3c-4e01-bb71-60fceaac3fe0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:52.656337 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe572b39-0b3c-4e01-bb71-60fceaac3fe0 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:52.674378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 733/1449] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:54.949706 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea4139f-597c-49d5-a0f5-a69ebe8e8eb6 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:54.950291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea4139f-597c-49d5-a0f5-a69ebe8e8eb6 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:54.950291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea4139f-597c-49d5-a0f5-a69ebe8e8eb6 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:54.951596 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 717/1450] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:08:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:54.956908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5ec79-ac83-4421-8d20-e82c001aee1b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:54.957313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5ec79-ac83-4421-8d20-e82c001aee1b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:54.957456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5ec79-ac83-4421-8d20-e82c001aee1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:55.022123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-0ea5ec79-ac83-4421-8d20-e82c001aee1b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:55.023322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0ea5ec79-ac83-4421-8d20-e82c001aee1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:55.043747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:55.117901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 734/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:08:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:08:55.414856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea4139f-597c-49d5-a0f5-a69ebe8e8eb6 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:55.416468 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0ea4139f-597c-49d5-a0f5-a69ebe8e8eb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:55.518635 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea4139f-597c-49d5-a0f5-a69ebe8e8eb6 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:55.537473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:55.591356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e203b7-70cf-4e1a-b649-28544c8727bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:55.591955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e203b7-70cf-4e1a-b649-28544c8727bf service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:55.592233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e203b7-70cf-4e1a-b649-28544c8727bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:55.592801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e203b7-70cf-4e1a-b649-28544c8727bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:55.594367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b5e203b7-70cf-4e1a-b649-28544c8727bf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:55.705975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e203b7-70cf-4e1a-b649-28544c8727bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:55.707335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e203b7-70cf-4e1a-b649-28544c8727bf service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:55.725392 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 718/1452] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:08:57.950918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea2b8ef-48f8-4221-8552-39123698599b None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:57.951493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea2b8ef-48f8-4221-8552-39123698599b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:57.951493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea2b8ef-48f8-4221-8552-39123698599b None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:57.952878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 735/1453] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:08:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:08:57.958351 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4a2594-6e24-43b3-ae8c-756bdb9f3ae9 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:57.958785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4a2594-6e24-43b3-ae8c-756bdb9f3ae9 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:57.958896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4a2594-6e24-43b3-ae8c-756bdb9f3ae9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:58.021304 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:58.024101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3c4a2594-6e24-43b3-ae8c-756bdb9f3ae9 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:08:58.025997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3c4a2594-6e24-43b3-ae8c-756bdb9f3ae9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:58.121862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 719/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:08:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:08:58.407890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea2b8ef-48f8-4221-8552-39123698599b None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:08:58.409470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7ea2b8ef-48f8-4221-8552-39123698599b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:58.513343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea2b8ef-48f8-4221-8552-39123698599b None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:08:58.562472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:08:58.586124 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029980e1-12fe-401f-b876-a2f0991ec1dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:08:58.586718 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029980e1-12fe-401f-b876-a2f0991ec1dc service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:08:58.586992 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029980e1-12fe-401f-b876-a2f0991ec1dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:08:58.587262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029980e1-12fe-401f-b876-a2f0991ec1dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:08:58.589380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-029980e1-12fe-401f-b876-a2f0991ec1dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:08:58.694478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029980e1-12fe-401f-b876-a2f0991ec1dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:08:58.695860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029980e1-12fe-401f-b876-a2f0991ec1dc service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:08:58.714281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 736/1455] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:08:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:09:00.927009 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bb17c6-941c-4008-82ae-bb735b9e620b None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:00.927579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bb17c6-941c-4008-82ae-bb735b9e620b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:00.927579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bb17c6-941c-4008-82ae-bb735b9e620b None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:00.928933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 720/1456] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:09:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:09:00.934461 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3181dc-5d69-4325-a9d1-2ecfc6433a25 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:00.934894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3181dc-5d69-4325-a9d1-2ecfc6433a25 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:00.935011 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3181dc-5d69-4325-a9d1-2ecfc6433a25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:01.000169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-ec3181dc-5d69-4325-a9d1-2ecfc6433a25 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:09:01.001439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ec3181dc-5d69-4325-a9d1-2ecfc6433a25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:01.034271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:01.103031 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 737/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:09:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:09:01.403188 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bb17c6-941c-4008-82ae-bb735b9e620b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:09:01.404790 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-72bb17c6-941c-4008-82ae-bb735b9e620b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:01.507104 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bb17c6-941c-4008-82ae-bb735b9e620b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:09:01.534805 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:01.578823 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59818b70-c3a6-49be-a44c-ea08abd2583d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:09:01.579416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59818b70-c3a6-49be-a44c-ea08abd2583d service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:01.579690 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59818b70-c3a6-49be-a44c-ea08abd2583d service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:01.579958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59818b70-c3a6-49be-a44c-ea08abd2583d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:01.581558 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-59818b70-c3a6-49be-a44c-ea08abd2583d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:01.686153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59818b70-c3a6-49be-a44c-ea08abd2583d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:09:01.687532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59818b70-c3a6-49be-a44c-ea08abd2583d service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:09:01.710810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 721/1458] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:09:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:09:03.930000 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22f8652-d27d-4419-a70f-aaf23f5e2001 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:03.930576 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22f8652-d27d-4419-a70f-aaf23f5e2001 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:03.930576 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22f8652-d27d-4419-a70f-aaf23f5e2001 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:03.932550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 738/1459] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:09:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:09:03.938070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fc2072-6b40-4a1f-985f-f7e5bf791671 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:03.938532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fc2072-6b40-4a1f-985f-f7e5bf791671 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:03.938631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fc2072-6b40-4a1f-985f-f7e5bf791671 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:04.004980 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-31fc2072-6b40-4a1f-985f-f7e5bf791671 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:09:04.006167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-31fc2072-6b40-4a1f-985f-f7e5bf791671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:04.014248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:04.107237 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 722/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:09:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:09:04.391702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22f8652-d27d-4419-a70f-aaf23f5e2001 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:09:04.393283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c22f8652-d27d-4419-a70f-aaf23f5e2001 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:04.496421 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22f8652-d27d-4419-a70f-aaf23f5e2001 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:09:04.550608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:04.568231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a203de81-a2b5-4b50-94b2-2043751a6c4f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:09:04.568857 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a203de81-a2b5-4b50-94b2-2043751a6c4f service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:04.569134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a203de81-a2b5-4b50-94b2-2043751a6c4f service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:04.569405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a203de81-a2b5-4b50-94b2-2043751a6c4f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:04.570991 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a203de81-a2b5-4b50-94b2-2043751a6c4f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:04.678049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a203de81-a2b5-4b50-94b2-2043751a6c4f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:09:04.679462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a203de81-a2b5-4b50-94b2-2043751a6c4f service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:09:04.697920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 739/1461] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:09:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:09:06.938558 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f8a87f-48b4-48f8-8ee5-f185e214aa10 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:06.939135 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f8a87f-48b4-48f8-8ee5-f185e214aa10 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:06.939135 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f8a87f-48b4-48f8-8ee5-f185e214aa10 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:06.940486 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 723/1462] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:09:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:09:06.945953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe0721b-0f92-4858-98a8-26610b271501 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:06.946890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe0721b-0f92-4858-98a8-26610b271501 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:06.947017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe0721b-0f92-4858-98a8-26610b271501 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:07.013011 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-0fe0721b-0f92-4858-98a8-26610b271501 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:09:07.014213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0fe0721b-0f92-4858-98a8-26610b271501 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:07.052026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:07.110240 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 740/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:09:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:09:07.395955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f8a87f-48b4-48f8-8ee5-f185e214aa10 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:09:07.397527 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b8f8a87f-48b4-48f8-8ee5-f185e214aa10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:07.501418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f8a87f-48b4-48f8-8ee5-f185e214aa10 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:09:07.539835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:07.575346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29407004-c8d7-4053-a91a-bd98959e59a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:09:07.575942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29407004-c8d7-4053-a91a-bd98959e59a2 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:07.576222 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29407004-c8d7-4053-a91a-bd98959e59a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:07.576524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29407004-c8d7-4053-a91a-bd98959e59a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:07.578105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-29407004-c8d7-4053-a91a-bd98959e59a2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:07.684848 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29407004-c8d7-4053-a91a-bd98959e59a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:09:07.686252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29407004-c8d7-4053-a91a-bd98959e59a2 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:09:07.704141 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 724/1464] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:09:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:09:09.872856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054c2126-8d65-493f-910b-64bff72d47d5 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:09.873403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054c2126-8d65-493f-910b-64bff72d47d5 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:09.873403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054c2126-8d65-493f-910b-64bff72d47d5 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:09.874736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 741/1465] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:09:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:09:09.880149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f510603-b92b-4781-ac64-ab2736112735 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:09.880627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f510603-b92b-4781-ac64-ab2736112735 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:09.880724 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f510603-b92b-4781-ac64-ab2736112735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:09.945697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-6f510603-b92b-4781-ac64-ab2736112735 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:09:09.947417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6f510603-b92b-4781-ac64-ab2736112735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:09.965880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:10.043631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 725/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:09:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:09:10.327147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054c2126-8d65-493f-910b-64bff72d47d5 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:09:10.328702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-054c2126-8d65-493f-910b-64bff72d47d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:10.430053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054c2126-8d65-493f-910b-64bff72d47d5 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:09:10.497339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:10.504562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc26718-974d-443f-becc-ef9a1e637c2a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:09:10.505160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc26718-974d-443f-becc-ef9a1e637c2a service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:10.505436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc26718-974d-443f-becc-ef9a1e637c2a service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:10.505703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc26718-974d-443f-becc-ef9a1e637c2a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:10.507324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-dbc26718-974d-443f-becc-ef9a1e637c2a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:10.616892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc26718-974d-443f-becc-ef9a1e637c2a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:09:10.618270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc26718-974d-443f-becc-ef9a1e637c2a service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:09:10.636813 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 742/1467] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:09:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:09:12.817329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84676993-aef4-4c54-9ab2-fd3699852b1b None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:12.817882 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84676993-aef4-4c54-9ab2-fd3699852b1b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:12.817882 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84676993-aef4-4c54-9ab2-fd3699852b1b None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:12.819252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 726/1468] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:09:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:09:12.825089 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76357672-2046-44a6-b053-2146b9c6168b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:12.825535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76357672-2046-44a6-b053-2146b9c6168b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:12.825640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76357672-2046-44a6-b053-2146b9c6168b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:12.892235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-76357672-2046-44a6-b053-2146b9c6168b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:09:12.893471 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-76357672-2046-44a6-b053-2146b9c6168b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:12.940863 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:12.988954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 743/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:09:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:09:13.270027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84676993-aef4-4c54-9ab2-fd3699852b1b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:09:13.271571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-84676993-aef4-4c54-9ab2-fd3699852b1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:13.372863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84676993-aef4-4c54-9ab2-fd3699852b1b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:09:13.420567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:13.445797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cc244d-38d4-49ff-8f5b-40275e8ba974 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:09:13.446412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc244d-38d4-49ff-8f5b-40275e8ba974 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:13.446694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc244d-38d4-49ff-8f5b-40275e8ba974 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:13.446959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc244d-38d4-49ff-8f5b-40275e8ba974 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:13.448557 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-19cc244d-38d4-49ff-8f5b-40275e8ba974 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:13.555008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cc244d-38d4-49ff-8f5b-40275e8ba974 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:09:13.556520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cc244d-38d4-49ff-8f5b-40275e8ba974 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:09:13.575440 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 727/1470] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:09:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:09:22.987429 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=22 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:23.572323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=66 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:33.494336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36897574-7abd-4481-b42f-ba751a4b154b None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:33.494865 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36897574-7abd-4481-b42f-ba751a4b154b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:33.494865 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36897574-7abd-4481-b42f-ba751a4b154b None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:33.496667 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 744/1471] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:09:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:09:33.501753 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6c4127-05b0-43f5-bb1e-4d91b8d04295 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:33.502154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6c4127-05b0-43f5-bb1e-4d91b8d04295 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:33.502308 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6c4127-05b0-43f5-bb1e-4d91b8d04295 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:33.566441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-eb6c4127-05b0-43f5-bb1e-4d91b8d04295 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:09:33.567633 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-eb6c4127-05b0-43f5-bb1e-4d91b8d04295 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:33.660486 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 728/1472] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:09:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:09:33.960266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5057e203-d80d-495a-b5dc-b6c3139fbbad None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:33.960753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5057e203-d80d-495a-b5dc-b6c3139fbbad None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:33.960972 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5057e203-d80d-495a-b5dc-b6c3139fbbad None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:33.962981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 745/1473] 10.222.0.26 () {54 vars in 962 bytes} [Wed Sep 28 12:09:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:09:33.969281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c4bdfb-8900-4f22-82c0-a67760c883f7 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:33.969571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c4bdfb-8900-4f22-82c0-a67760c883f7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:33.969842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c4bdfb-8900-4f22-82c0-a67760c883f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:34.044961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-69c4bdfb-8900-4f22-82c0-a67760c883f7 None None] MFA Rules not processed for user `815504693bcc40b6aef06aacb2a9a35b`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:09:34.046177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-69c4bdfb-8900-4f22-82c0-a67760c883f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:34.066627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:34.121518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 729/1474] 10.222.0.26 () {58 vars in 1059 bytes} [Wed Sep 28 12:09:33 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:09:34.129620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5057e203-d80d-495a-b5dc-b6c3139fbbad None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:09:34.131102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5057e203-d80d-495a-b5dc-b6c3139fbbad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:34.235269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5057e203-d80d-495a-b5dc-b6c3139fbbad None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:09:34.307436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc5c73a-75c1-48e0-8b9b-c6121302db9d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '815504693bcc40b6aef06aacb2a9a35b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:09:34.308034 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc5c73a-75c1-48e0-8b9b-c6121302db9d service glance] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:34.308343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc5c73a-75c1-48e0-8b9b-c6121302db9d service glance] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:34.308619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc5c73a-75c1-48e0-8b9b-c6121302db9d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:34.310194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4bc5c73a-75c1-48e0-8b9b-c6121302db9d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:34.364249 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:34.419368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc5c73a-75c1-48e0-8b9b-c6121302db9d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:09:34.420822 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc5c73a-75c1-48e0-8b9b-c6121302db9d service glance] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:09:34.438553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 746/1475] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:09:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:09:36.682887 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff1ad65-f4f9-43f8-8ed5-a659c61c2382 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:36.683420 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff1ad65-f4f9-43f8-8ed5-a659c61c2382 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:36.683420 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff1ad65-f4f9-43f8-8ed5-a659c61c2382 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:36.684815 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 730/1476] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:09:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:09:36.690215 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ef0776-01c2-4cb6-af7b-26278add07fd None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:36.690620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ef0776-01c2-4cb6-af7b-26278add07fd None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:36.690767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ef0776-01c2-4cb6-af7b-26278add07fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:36.755134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-17ef0776-01c2-4cb6-af7b-26278add07fd None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:09:36.756370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-17ef0776-01c2-4cb6-af7b-26278add07fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:36.868173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 747/1477] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:09:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:09:37.154896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff1ad65-f4f9-43f8-8ed5-a659c61c2382 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:09:37.156536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3ff1ad65-f4f9-43f8-8ed5-a659c61c2382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:37.280556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff1ad65-f4f9-43f8-8ed5-a659c61c2382 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:09:37.329201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:37.372906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7d3997-de5d-43a8-84bf-dab38ac2aa4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:09:37.373503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7d3997-de5d-43a8-84bf-dab38ac2aa4a admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:37.373781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7d3997-de5d-43a8-84bf-dab38ac2aa4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:37.374051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7d3997-de5d-43a8-84bf-dab38ac2aa4a admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:37.374681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7d3997-de5d-43a8-84bf-dab38ac2aa4a admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:09:37.376512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: WARNING py.warnings [None req-7f7d3997-de5d-43a8-84bf-dab38ac2aa4a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:09:37.376512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: warnings.warn(msg) Sep 28 12:09:37.376512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: Sep 28 12:09:37.377666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7d3997-de5d-43a8-84bf-dab38ac2aa4a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:09:37.396831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 731/1478] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Sep 28 12:09:37 2022] GET /identity/v3/projects => generated 2553 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:09:39.583977 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b466ce-dd7c-4e3a-8ca3-ae1e852be234 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:39.584545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b466ce-dd7c-4e3a-8ca3-ae1e852be234 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:39.584545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b466ce-dd7c-4e3a-8ca3-ae1e852be234 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:39.586402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 748/1479] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:09:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:09:39.591983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ee74a2-5c61-4921-9010-3f43bd13bb6c None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:39.592730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ee74a2-5c61-4921-9010-3f43bd13bb6c None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:39.592837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ee74a2-5c61-4921-9010-3f43bd13bb6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:39.658354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-45ee74a2-5c61-4921-9010-3f43bd13bb6c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:09:39.659494 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-45ee74a2-5c61-4921-9010-3f43bd13bb6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:39.754160 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 732/1480] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:09:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:09:40.049352 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b466ce-dd7c-4e3a-8ca3-ae1e852be234 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:09:40.050861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a8b466ce-dd7c-4e3a-8ca3-ae1e852be234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:40.061046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:40.195487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b466ce-dd7c-4e3a-8ca3-ae1e852be234 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:09:40.273860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:40.306054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a0582e-7bc2-4fed-b672-d3f795c990b0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:09:40.306561 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a0582e-7bc2-4fed-b672-d3f795c990b0 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:40.306835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a0582e-7bc2-4fed-b672-d3f795c990b0 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:40.307104 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a0582e-7bc2-4fed-b672-d3f795c990b0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:40.308689 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-46a0582e-7bc2-4fed-b672-d3f795c990b0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:40.396165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a0582e-7bc2-4fed-b672-d3f795c990b0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:09:40.397763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a0582e-7bc2-4fed-b672-d3f795c990b0 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:09:40.416074 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 749/1481] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:09:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:09:42.746301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb06ab3b-227d-4fe0-ba8c-cc158c4ff314 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:42.746818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb06ab3b-227d-4fe0-ba8c-cc158c4ff314 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:42.746818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb06ab3b-227d-4fe0-ba8c-cc158c4ff314 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:42.748118 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 733/1482] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:09:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:09:42.753410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184f6bb6-362e-40e4-b50e-0ea0ff1e627e None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:42.753793 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184f6bb6-362e-40e4-b50e-0ea0ff1e627e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:42.753946 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184f6bb6-362e-40e4-b50e-0ea0ff1e627e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:42.817287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-184f6bb6-362e-40e4-b50e-0ea0ff1e627e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:09:42.818472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-184f6bb6-362e-40e4-b50e-0ea0ff1e627e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:42.866451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:42.916911 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 750/1483] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:09:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:09:43.207388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb06ab3b-227d-4fe0-ba8c-cc158c4ff314 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:09:43.208987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-cb06ab3b-227d-4fe0-ba8c-cc158c4ff314 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:43.232580 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:43.309062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb06ab3b-227d-4fe0-ba8c-cc158c4ff314 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:09:43.380588 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc97bc8f-38e1-4d50-bb53-8476ab03df69 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:09:43.381180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc97bc8f-38e1-4d50-bb53-8476ab03df69 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:43.381457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc97bc8f-38e1-4d50-bb53-8476ab03df69 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:43.381724 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc97bc8f-38e1-4d50-bb53-8476ab03df69 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:43.383879 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-bc97bc8f-38e1-4d50-bb53-8476ab03df69 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:43.454707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:43.489827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc97bc8f-38e1-4d50-bb53-8476ab03df69 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:09:43.491283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc97bc8f-38e1-4d50-bb53-8476ab03df69 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:09:43.510313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 734/1484] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:09:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:09:45.702604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5705529e-2ea8-4691-9319-a48651f7b132 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:45.703129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5705529e-2ea8-4691-9319-a48651f7b132 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:45.703129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5705529e-2ea8-4691-9319-a48651f7b132 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:45.704475 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 751/1485] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:09:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:09:45.710102 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cbdc6f-f120-425c-8cd8-72eecd68cfe0 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:45.710615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cbdc6f-f120-425c-8cd8-72eecd68cfe0 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:45.710717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cbdc6f-f120-425c-8cd8-72eecd68cfe0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:45.775960 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-c7cbdc6f-f120-425c-8cd8-72eecd68cfe0 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:09:45.777205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c7cbdc6f-f120-425c-8cd8-72eecd68cfe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:45.900938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 735/1486] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:09:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:09:46.188197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5705529e-2ea8-4691-9319-a48651f7b132 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:09:46.189798 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5705529e-2ea8-4691-9319-a48651f7b132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:46.291656 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5705529e-2ea8-4691-9319-a48651f7b132 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:09:46.361908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:46.363732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5ae046-78fd-4e3f-aa03-aa1f000e6170 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:09:46.364364 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5ae046-78fd-4e3f-aa03-aa1f000e6170 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:46.364844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5ae046-78fd-4e3f-aa03-aa1f000e6170 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:46.365277 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5ae046-78fd-4e3f-aa03-aa1f000e6170 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:46.367594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ec5ae046-78fd-4e3f-aa03-aa1f000e6170 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:46.478255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5ae046-78fd-4e3f-aa03-aa1f000e6170 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:09:46.479598 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5ae046-78fd-4e3f-aa03-aa1f000e6170 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:09:46.498337 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 752/1487] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:09:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:09:48.744036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb899539-33f9-476c-8c73-b8e0af218478 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:48.744620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb899539-33f9-476c-8c73-b8e0af218478 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:48.744620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb899539-33f9-476c-8c73-b8e0af218478 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:48.745898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 736/1488] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:09:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:09:48.752034 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef630b5-1e2a-4f56-91b1-0abcf4a324ca None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:48.752464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef630b5-1e2a-4f56-91b1-0abcf4a324ca None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:48.752605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef630b5-1e2a-4f56-91b1-0abcf4a324ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:48.816782 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-fef630b5-1e2a-4f56-91b1-0abcf4a324ca None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:09:48.817979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-fef630b5-1e2a-4f56-91b1-0abcf4a324ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:48.870296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:48.919711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 753/1489] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:09:49.212851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb899539-33f9-476c-8c73-b8e0af218478 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:09:49.214382 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-bb899539-33f9-476c-8c73-b8e0af218478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:49.250169 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:49.316437 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb899539-33f9-476c-8c73-b8e0af218478 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:09:49.389318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52aed8cd-5802-4d76-8ea4-e275591ee32c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:09:49.389913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aed8cd-5802-4d76-8ea4-e275591ee32c service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:49.390192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aed8cd-5802-4d76-8ea4-e275591ee32c service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:49.390462 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aed8cd-5802-4d76-8ea4-e275591ee32c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:49.392035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-52aed8cd-5802-4d76-8ea4-e275591ee32c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:49.587554 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:49.618192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52aed8cd-5802-4d76-8ea4-e275591ee32c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:09:49.619799 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52aed8cd-5802-4d76-8ea4-e275591ee32c service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:09:49.637526 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 737/1490] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:09:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:09:51.995841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dce9c02-ce8f-4663-9a7a-0208a5c6aa1e None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:51.996412 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dce9c02-ce8f-4663-9a7a-0208a5c6aa1e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:51.996412 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dce9c02-ce8f-4663-9a7a-0208a5c6aa1e None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:51.998116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 754/1491] 10.222.0.26 () {54 vars in 927 bytes} [Wed Sep 28 12:09:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:09:52.003280 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae47052e-2c1a-4435-add9-d201a0e33525 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:52.003714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae47052e-2c1a-4435-add9-d201a0e33525 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:52.003825 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae47052e-2c1a-4435-add9-d201a0e33525 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:52.068408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-ae47052e-2c1a-4435-add9-d201a0e33525 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:09:52.070151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-ae47052e-2c1a-4435-add9-d201a0e33525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:52.164152 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 738/1492] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Sep 28 12:09:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:09:52.454702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dce9c02-ce8f-4663-9a7a-0208a5c6aa1e None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:09:52.456216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5dce9c02-ce8f-4663-9a7a-0208a5c6aa1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:52.571532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dce9c02-ce8f-4663-9a7a-0208a5c6aa1e None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:09:52.643903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525c6bd0-ee4b-4ed8-a667-b676a28febfe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:09:52.644604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525c6bd0-ee4b-4ed8-a667-b676a28febfe service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:09:52.644888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525c6bd0-ee4b-4ed8-a667-b676a28febfe service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:09:52.645161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525c6bd0-ee4b-4ed8-a667-b676a28febfe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:09:52.646750 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-525c6bd0-ee4b-4ed8-a667-b676a28febfe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:09:52.655627 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:09:52.753260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525c6bd0-ee4b-4ed8-a667-b676a28febfe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:09:52.754592 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525c6bd0-ee4b-4ed8-a667-b676a28febfe service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:09:52.773325 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 755/1493] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:09:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:10:02.162614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=82 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:10:02.770322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=50 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:11:11.063063 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea11544-616d-4d2c-b92a-bdbcc664af00 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:11.063664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea11544-616d-4d2c-b92a-bdbcc664af00 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:11.063664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea11544-616d-4d2c-b92a-bdbcc664af00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:11.140342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-dea11544-616d-4d2c-b92a-bdbcc664af00 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:11:11.140342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-dea11544-616d-4d2c-b92a-bdbcc664af00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:11.234933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 739/1494] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:11:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:11:11.243029 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5a3761-9770-4d6d-a158-e683fb2746a6 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:11.243537 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5a3761-9770-4d6d-a158-e683fb2746a6 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:11.243658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5a3761-9770-4d6d-a158-e683fb2746a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:11.317917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4e5a3761-9770-4d6d-a158-e683fb2746a6 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:11:11.319041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4e5a3761-9770-4d6d-a158-e683fb2746a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:11.414688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 756/1495] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:11:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:11:11.419470 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea11544-616d-4d2c-b92a-bdbcc664af00 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:11:11.420711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-dea11544-616d-4d2c-b92a-bdbcc664af00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:11.432181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea11544-616d-4d2c-b92a-bdbcc664af00 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:11:11.494946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:11:11.528792 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff988da-f98c-4541-8560-59316b0092e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:11:11.529387 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff988da-f98c-4541-8560-59316b0092e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:11.529661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff988da-f98c-4541-8560-59316b0092e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:11.529925 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff988da-f98c-4541-8560-59316b0092e1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:11.530560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff988da-f98c-4541-8560-59316b0092e1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:11:11.530855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff988da-f98c-4541-8560-59316b0092e1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:11:11.531858 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: WARNING py.warnings [None req-fff988da-f98c-4541-8560-59316b0092e1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:11:11.531858 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: warnings.warn(msg) Sep 28 12:11:11.531858 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: Sep 28 12:11:11.532767 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff988da-f98c-4541-8560-59316b0092e1 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:11:11.544719 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 740/1496] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:11:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:11:11.553650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5a3761-9770-4d6d-a158-e683fb2746a6 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:11:11.562802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5a3761-9770-4d6d-a158-e683fb2746a6 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:11:11.623725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:11:11.655836 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6540b0-57de-4d56-be7f-4e115b51a719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:11:11.656457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6540b0-57de-4d56-be7f-4e115b51a719 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:11.656734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6540b0-57de-4d56-be7f-4e115b51a719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:11.657011 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6540b0-57de-4d56-be7f-4e115b51a719 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:11.657641 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6540b0-57de-4d56-be7f-4e115b51a719 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:11:11.659340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: WARNING py.warnings [None req-6c6540b0-57de-4d56-be7f-4e115b51a719 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:11:11.659340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: warnings.warn(msg) Sep 28 12:11:11.659340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: Sep 28 12:11:11.661065 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6540b0-57de-4d56-be7f-4e115b51a719 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:11:11.715058 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 757/1497] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:11:11 2022] POST /identity/v3/projects => generated 375 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:11:11.719494 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff988da-f98c-4541-8560-59316b0092e1 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:11:11.721014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-fff988da-f98c-4541-8560-59316b0092e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:11.753937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff988da-f98c-4541-8560-59316b0092e1 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:11:11.845123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721e692f-012b-4df6-847b-dc5e10a2126d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:11:11.845721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721e692f-012b-4df6-847b-dc5e10a2126d admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:11.845998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721e692f-012b-4df6-847b-dc5e10a2126d admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:11.846266 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721e692f-012b-4df6-847b-dc5e10a2126d admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:11.846900 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721e692f-012b-4df6-847b-dc5e10a2126d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:11:11.848484 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: WARNING py.warnings [None req-721e692f-012b-4df6-847b-dc5e10a2126d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:11:11.848484 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: warnings.warn(msg) Sep 28 12:11:11.848484 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: Sep 28 12:11:11.849695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721e692f-012b-4df6-847b-dc5e10a2126d admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:11:11.890742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 741/1498] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:11:11 2022] POST /identity/v3/users => generated 288 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:11:11.895259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6540b0-57de-4d56-be7f-4e115b51a719 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:11:11.896778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6c6540b0-57de-4d56-be7f-4e115b51a719 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:11.957782 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6540b0-57de-4d56-be7f-4e115b51a719 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:11:12.041481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:11:12.052759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054616e2-2699-474d-8a69-ed9dc58042bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:11:12.053361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054616e2-2699-474d-8a69-ed9dc58042bd admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:12.053635 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054616e2-2699-474d-8a69-ed9dc58042bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:12.053903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054616e2-2699-474d-8a69-ed9dc58042bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:12.054548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054616e2-2699-474d-8a69-ed9dc58042bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:11:12.055535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: WARNING py.warnings [None req-054616e2-2699-474d-8a69-ed9dc58042bd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:11:12.055535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: warnings.warn(msg) Sep 28 12:11:12.055535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: Sep 28 12:11:12.056437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054616e2-2699-474d-8a69-ed9dc58042bd admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:11:12.070661 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 758/1499] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:11:11 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:11:12.074723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721e692f-012b-4df6-847b-dc5e10a2126d admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:11:12.076234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-721e692f-012b-4df6-847b-dc5e10a2126d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:12.088113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721e692f-012b-4df6-847b-dc5e10a2126d admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:11:12.113596 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:11:12.182398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44301e6-1fef-42ee-8673-ef4a691db2b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:11:12.182940 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44301e6-1fef-42ee-8673-ef4a691db2b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:12.183219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44301e6-1fef-42ee-8673-ef4a691db2b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:12.183490 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44301e6-1fef-42ee-8673-ef4a691db2b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:12.184131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44301e6-1fef-42ee-8673-ef4a691db2b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:11:12.185187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: WARNING py.warnings [None req-e44301e6-1fef-42ee-8673-ef4a691db2b2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:11:12.185187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: warnings.warn(msg) Sep 28 12:11:12.185187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: Sep 28 12:11:12.186080 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44301e6-1fef-42ee-8673-ef4a691db2b2 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:11:12.198672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 742/1500] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:11:12 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:11:12.202822 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054616e2-2699-474d-8a69-ed9dc58042bd admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:11:12.204424 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-054616e2-2699-474d-8a69-ed9dc58042bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:12.237283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054616e2-2699-474d-8a69-ed9dc58042bd admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:11:12.329413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b5ada1-1317-4985-bce7-45d26d4c77a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:11:12.330091 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b5ada1-1317-4985-bce7-45d26d4c77a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:12.330321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b5ada1-1317-4985-bce7-45d26d4c77a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:12.330591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b5ada1-1317-4985-bce7-45d26d4c77a5 admin admin] PATH_INFO: `/v3/projects/e45461dde209418b9bb6cf00fdba7cfe/users/26512072376042f4953b42355322449c/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:12.353193 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b5ada1-1317-4985-bce7-45d26d4c77a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e45461dde209418b9bb6cf00fdba7cfe, user_id=26512072376042f4953b42355322449c, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:11:12.354205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: WARNING py.warnings [None req-c0b5ada1-1317-4985-bce7-45d26d4c77a5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:11:12.354205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: warnings.warn(msg) Sep 28 12:11:12.354205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: Sep 28 12:11:12.356125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b5ada1-1317-4985-bce7-45d26d4c77a5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:11:12.380615 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 759/1501] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:11:12 2022] PUT /identity/v3/projects/e45461dde209418b9bb6cf00fdba7cfe/users/26512072376042f4953b42355322449c/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:11:12.392744 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644943c3-db5a-47e9-bef7-1dfa658810e3 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:12.393030 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644943c3-db5a-47e9-bef7-1dfa658810e3 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:12.393304 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644943c3-db5a-47e9-bef7-1dfa658810e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:12.448376 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-644943c3-db5a-47e9-bef7-1dfa658810e3 None None] MFA Rules not processed for user `26512072376042f4953b42355322449c`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:11:12.449533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-644943c3-db5a-47e9-bef7-1dfa658810e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:12.529202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 743/1502] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:11:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:11:12.542298 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40563a86-f194-4158-a0a0-c57a5dc734d4 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:12.542582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40563a86-f194-4158-a0a0-c57a5dc734d4 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:12.542856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40563a86-f194-4158-a0a0-c57a5dc734d4 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:12.544177 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 760/1503] 10.222.0.26 () {54 vars in 963 bytes} [Wed Sep 28 12:11:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:11:12.548996 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5079eae9-497b-4c15-8601-0042891faf62 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:12.549276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5079eae9-497b-4c15-8601-0042891faf62 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:12.549542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5079eae9-497b-4c15-8601-0042891faf62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:12.580754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:11:12.623676 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-5079eae9-497b-4c15-8601-0042891faf62 None None] MFA Rules not processed for user `2ba80861b08940d398a61879bb70060a`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:11:12.624885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5079eae9-497b-4c15-8601-0042891faf62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:12.697554 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 744/1504] 10.222.0.26 () {58 vars in 1060 bytes} [Wed Sep 28 12:11:12 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:11:12.705623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40563a86-f194-4158-a0a0-c57a5dc734d4 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:11:12.707102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-40563a86-f194-4158-a0a0-c57a5dc734d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:12.756476 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:11:12.810012 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40563a86-f194-4158-a0a0-c57a5dc734d4 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:11:12.885455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199d3bc2-d38f-44df-b7af-6d43dfb97f96 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:11:12.886055 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199d3bc2-d38f-44df-b7af-6d43dfb97f96 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:12.886332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199d3bc2-d38f-44df-b7af-6d43dfb97f96 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:12.886602 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199d3bc2-d38f-44df-b7af-6d43dfb97f96 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:12.888185 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-199d3bc2-d38f-44df-b7af-6d43dfb97f96 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:12.985218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199d3bc2-d38f-44df-b7af-6d43dfb97f96 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:11:12.986559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199d3bc2-d38f-44df-b7af-6d43dfb97f96 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:11:12.990044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:11:13.005175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 761/1505] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:11:12 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:11:13.050327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4267a2fa-6fa8-4c83-9c6f-42fe51f0d99a None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:13.050615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4267a2fa-6fa8-4c83-9c6f-42fe51f0d99a None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:13.050917 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4267a2fa-6fa8-4c83-9c6f-42fe51f0d99a None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:13.052178 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 745/1506] 10.222.0.26 () {52 vars in 830 bytes} [Wed Sep 28 12:11:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:11:13.055734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199d3bc2-d38f-44df-b7af-6d43dfb97f96 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:11:13.056992 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-199d3bc2-d38f-44df-b7af-6d43dfb97f96 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:13.154445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199d3bc2-d38f-44df-b7af-6d43dfb97f96 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:11:13.246216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:11:13.246766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:13.247041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:13.247309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] PATH_INFO: `/v3/users/26512072376042f4953b42355322449c` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:13.260219 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] RBAC: Authorizing `identity:delete_user(user_id=26512072376042f4953b42355322449c)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:11:13.261284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: WARNING py.warnings [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:11:13.261284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: warnings.warn(msg) Sep 28 12:11:13.261284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: Sep 28 12:11:13.262488 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:11:13.271981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:11:13.327879 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26512072376042f4953b42355322449c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:11:13.328443 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26512072376042f4953b42355322449c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:11:13.341812 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26512072376042f4953b42355322449c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:11:13.342640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26512072376042f4953b42355322449c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:11:13.365259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26512072376042f4953b42355322449c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:11:13.381920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 762/1507] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:11:13 2022] DELETE /identity/v3/users/26512072376042f4953b42355322449c => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:11:13.386464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b21367-2077-4eb2-b692-b853f20047e3 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:13.386952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b21367-2077-4eb2-b692-b853f20047e3 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:13.387061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b21367-2077-4eb2-b692-b853f20047e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:13.442910 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-25b21367-2077-4eb2-b692-b853f20047e3 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:11:13.444094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-25b21367-2077-4eb2-b692-b853f20047e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:13.523682 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:11:13.536095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 746/1508] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:11:13.545618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:11:13.546844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:13.643944 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca85f36-3d10-4bda-b79d-8ffdf2ecb489 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:11:13.698869 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:11:13.720064 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39dc4c4-ac85-49ea-bde0-30c239cd5e59 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:11:13.720702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39dc4c4-ac85-49ea-bde0-30c239cd5e59 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:13.720987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39dc4c4-ac85-49ea-bde0-30c239cd5e59 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:13.721255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39dc4c4-ac85-49ea-bde0-30c239cd5e59 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:13.722847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f39dc4c4-ac85-49ea-bde0-30c239cd5e59 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:13.834131 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39dc4c4-ac85-49ea-bde0-30c239cd5e59 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:11:13.835567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39dc4c4-ac85-49ea-bde0-30c239cd5e59 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:11:13.854336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 763/1509] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:11:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:11:14.202125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b21367-2077-4eb2-b692-b853f20047e3 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:11:14.203242 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-25b21367-2077-4eb2-b692-b853f20047e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:11:14.220408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b21367-2077-4eb2-b692-b853f20047e3 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:11:14.314607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04521f8-0cfe-4530-88a3-18fb5f3281c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:11:14.315232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04521f8-0cfe-4530-88a3-18fb5f3281c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:11:14.315508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04521f8-0cfe-4530-88a3-18fb5f3281c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:11:14.315776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04521f8-0cfe-4530-88a3-18fb5f3281c5 admin admin] PATH_INFO: `/v3/projects/e45461dde209418b9bb6cf00fdba7cfe` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:11:14.325491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04521f8-0cfe-4530-88a3-18fb5f3281c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e45461dde209418b9bb6cf00fdba7cfe)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:11:14.326511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: WARNING py.warnings [None req-e04521f8-0cfe-4530-88a3-18fb5f3281c5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:11:14.326511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: warnings.warn(msg) Sep 28 12:11:14.326511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: Sep 28 12:11:14.327713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04521f8-0cfe-4530-88a3-18fb5f3281c5 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:11:14.387001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e04521f8-0cfe-4530-88a3-18fb5f3281c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e45461dde209418b9bb6cf00fdba7cfe', 'request_id': 'req-e04521f8-0cfe-4530-88a3-18fb5f3281c5'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:11:14.392876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e04521f8-0cfe-4530-88a3-18fb5f3281c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e45461dde209418b9bb6cf00fdba7cfe', 'request_id': 'req-e04521f8-0cfe-4530-88a3-18fb5f3281c5'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:11:14.402387 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e04521f8-0cfe-4530-88a3-18fb5f3281c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e45461dde209418b9bb6cf00fdba7cfe', 'request_id': 'req-e04521f8-0cfe-4530-88a3-18fb5f3281c5'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:11:14.430412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e04521f8-0cfe-4530-88a3-18fb5f3281c5 admin admin] The token cache is being invalidate because project e45461dde209418b9bb6cf00fdba7cfe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:11:14.430696 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e04521f8-0cfe-4530-88a3-18fb5f3281c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:11:14.431201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e04521f8-0cfe-4530-88a3-18fb5f3281c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:11:14.432516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 747/1510] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:11:14 2022] DELETE /identity/v3/projects/e45461dde209418b9bb6cf00fdba7cfe => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:11:23.852040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=62 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:11:24.428518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:12:42.835540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631d1b29-af16-41c0-8930-b72cd05f3579 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:42.836188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631d1b29-af16-41c0-8930-b72cd05f3579 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:42.836188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631d1b29-af16-41c0-8930-b72cd05f3579 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:42.837491 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 764/1511] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 28 12:12:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:12:42.842174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b30c361-f3f9-40fd-b723-d1e9b58e513b None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:42.842871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b30c361-f3f9-40fd-b723-d1e9b58e513b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:42.842976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b30c361-f3f9-40fd-b723-d1e9b58e513b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:42.926425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-4b30c361-f3f9-40fd-b723-d1e9b58e513b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:12:42.927627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-4b30c361-f3f9-40fd-b723-d1e9b58e513b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:12:43.028833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 748/1512] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 28 12:12:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:12:43.182306 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631d1b29-af16-41c0-8930-b72cd05f3579 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:12:43.184498 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-631d1b29-af16-41c0-8930-b72cd05f3579 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:12:43.300844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631d1b29-af16-41c0-8930-b72cd05f3579 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:12:43.306411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f790986-24b7-4bec-8865-711abf5af39e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:12:43.307007 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f790986-24b7-4bec-8865-711abf5af39e service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:43.307284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f790986-24b7-4bec-8865-711abf5af39e service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:43.307557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f790986-24b7-4bec-8865-711abf5af39e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:43.316111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f790986-24b7-4bec-8865-711abf5af39e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:12:43.317731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f790986-24b7-4bec-8865-711abf5af39e service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:12:43.326928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 765/1513] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:12:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:12:44.434388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691b5181-5530-4dfb-9379-3aaa642581ad None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:44.434956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691b5181-5530-4dfb-9379-3aaa642581ad None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:44.434956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691b5181-5530-4dfb-9379-3aaa642581ad None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:44.436278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 749/1514] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 28 12:12:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:12:44.443721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d65fa10-0d1d-4ba6-b3f3-c7ff428f5196 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:44.444255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d65fa10-0d1d-4ba6-b3f3-c7ff428f5196 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:44.444600 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d65fa10-0d1d-4ba6-b3f3-c7ff428f5196 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:44.464261 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-1d65fa10-0d1d-4ba6-b3f3-c7ff428f5196 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:12:44.465424 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1d65fa10-0d1d-4ba6-b3f3-c7ff428f5196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:12:44.474092 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 766/1515] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 28 12:12:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:12:44.623558 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691b5181-5530-4dfb-9379-3aaa642581ad None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:12:44.637627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691b5181-5530-4dfb-9379-3aaa642581ad None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:12:44.643548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be074ae-3078-4dae-9d4d-e7a804d4b599 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:12:44.644132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be074ae-3078-4dae-9d4d-e7a804d4b599 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:44.644447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be074ae-3078-4dae-9d4d-e7a804d4b599 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:44.644735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be074ae-3078-4dae-9d4d-e7a804d4b599 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:44.652541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be074ae-3078-4dae-9d4d-e7a804d4b599 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:12:44.654108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be074ae-3078-4dae-9d4d-e7a804d4b599 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:12:44.663764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 750/1516] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:12:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:12:45.527435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6747ec-db2c-4796-a688-fa26e3670e6c None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:45.527889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6747ec-db2c-4796-a688-fa26e3670e6c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:45.527992 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6747ec-db2c-4796-a688-fa26e3670e6c None None] PATH_INFO: `/v3` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:45.529300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 767/1517] 10.222.0.25 () {54 vars in 925 bytes} [Wed Sep 28 12:12:45 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:12:45.535104 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e0d410-711b-41db-a38b-b0acb5d5e2fc None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:45.535537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e0d410-711b-41db-a38b-b0acb5d5e2fc None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:45.535652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e0d410-711b-41db-a38b-b0acb5d5e2fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:45.593600 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-c5e0d410-711b-41db-a38b-b0acb5d5e2fc None None] MFA Rules not processed for user `3758ea0f26264340ad2f154c0111af21`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:12:45.594863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c5e0d410-711b-41db-a38b-b0acb5d5e2fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:12:45.610977 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:12:45.666890 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 751/1518] 10.222.0.25 () {58 vars in 1015 bytes} [Wed Sep 28 12:12:45 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:12:49.389216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c02f07-492c-40a7-93b4-2d11d8fe209d None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:49.389656 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c02f07-492c-40a7-93b4-2d11d8fe209d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:49.389761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c02f07-492c-40a7-93b4-2d11d8fe209d None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:49.391099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 768/1519] 10.222.0.26 () {54 vars in 961 bytes} [Wed Sep 28 12:12:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:12:49.396044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c009bab-2f32-4645-afee-e09105d83c1e None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:49.396495 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c009bab-2f32-4645-afee-e09105d83c1e None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:49.396628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c009bab-2f32-4645-afee-e09105d83c1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:49.416185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-8c009bab-2f32-4645-afee-e09105d83c1e None None] MFA Rules not processed for user `3758ea0f26264340ad2f154c0111af21`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:12:49.417399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8c009bab-2f32-4645-afee-e09105d83c1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:12:49.426469 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 752/1520] 10.222.0.26 () {58 vars in 1058 bytes} [Wed Sep 28 12:12:49 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:12:49.433884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c02f07-492c-40a7-93b4-2d11d8fe209d None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:12:49.446909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c02f07-492c-40a7-93b4-2d11d8fe209d None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:12:49.451680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa0fe91-def4-4096-b9ee-26f05437a1c5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:12:49.452252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa0fe91-def4-4096-b9ee-26f05437a1c5 service placement] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:49.452556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa0fe91-def4-4096-b9ee-26f05437a1c5 service placement] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:49.452825 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa0fe91-def4-4096-b9ee-26f05437a1c5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:49.461076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa0fe91-def4-4096-b9ee-26f05437a1c5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:12:49.462750 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa0fe91-def4-4096-b9ee-26f05437a1c5 service placement] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:12:49.473089 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 769/1521] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:12:49 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:12:56.202616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587df905-9b77-49c8-9a72-8739620a597b None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:56.203221 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587df905-9b77-49c8-9a72-8739620a597b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:56.203221 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587df905-9b77-49c8-9a72-8739620a597b None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:56.204589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 753/1522] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 28 12:12:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:12:56.210781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb7e949-fce6-49a7-8f26-a15db20294d8 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:56.211210 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb7e949-fce6-49a7-8f26-a15db20294d8 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:56.211334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb7e949-fce6-49a7-8f26-a15db20294d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:56.232301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-beb7e949-fce6-49a7-8f26-a15db20294d8 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:12:56.233758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-beb7e949-fce6-49a7-8f26-a15db20294d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:12:56.242018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 770/1523] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 28 12:12:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:12:56.395740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587df905-9b77-49c8-9a72-8739620a597b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:12:56.411229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587df905-9b77-49c8-9a72-8739620a597b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:12:56.417841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dc43df-aa8a-4733-abf5-823e54433605 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:12:56.418500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dc43df-aa8a-4733-abf5-823e54433605 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:56.418797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dc43df-aa8a-4733-abf5-823e54433605 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:56.419230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dc43df-aa8a-4733-abf5-823e54433605 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:56.429257 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99dc43df-aa8a-4733-abf5-823e54433605 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:12:56.430857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99dc43df-aa8a-4733-abf5-823e54433605 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:12:56.440360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 754/1524] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:12:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:12:57.522381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5350fe-bdce-421e-a749-4b5b3a4d6893 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:57.522811 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5350fe-bdce-421e-a749-4b5b3a4d6893 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:57.522903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5350fe-bdce-421e-a749-4b5b3a4d6893 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:57.524252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 771/1525] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 28 12:12:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:12:57.530497 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9c5e73-8d9a-4fd5-b573-bdf7faee3a20 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:57.530958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9c5e73-8d9a-4fd5-b573-bdf7faee3a20 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:57.531066 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9c5e73-8d9a-4fd5-b573-bdf7faee3a20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:57.550986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-fa9c5e73-8d9a-4fd5-b573-bdf7faee3a20 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:12:57.552165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-fa9c5e73-8d9a-4fd5-b573-bdf7faee3a20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:12:57.559855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 755/1526] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 28 12:12:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:12:57.723262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5350fe-bdce-421e-a749-4b5b3a4d6893 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:12:57.737809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5350fe-bdce-421e-a749-4b5b3a4d6893 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:12:57.743392 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fec08d2-1a5e-4940-aff5-b84cf9762e7c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:12:57.744032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fec08d2-1a5e-4940-aff5-b84cf9762e7c service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:12:57.744254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fec08d2-1a5e-4940-aff5-b84cf9762e7c service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:12:57.744565 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fec08d2-1a5e-4940-aff5-b84cf9762e7c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:12:57.752447 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fec08d2-1a5e-4940-aff5-b84cf9762e7c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:12:57.754004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fec08d2-1a5e-4940-aff5-b84cf9762e7c service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:12:57.763043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 772/1527] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:12:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:13:01.731282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220019d2-2549-40fd-98f6-6f0e484c0248 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:13:01.731865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220019d2-2549-40fd-98f6-6f0e484c0248 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:13:01.731865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220019d2-2549-40fd-98f6-6f0e484c0248 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:13:01.733197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 756/1528] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:13:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:13:01.738845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5f8dbd-2040-4b57-9232-2dd52554a4f5 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:13:01.739273 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5f8dbd-2040-4b57-9232-2dd52554a4f5 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:13:01.739382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5f8dbd-2040-4b57-9232-2dd52554a4f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:13:01.759412 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4f5f8dbd-2040-4b57-9232-2dd52554a4f5 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:13:01.760583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4f5f8dbd-2040-4b57-9232-2dd52554a4f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:13:01.768353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 773/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:13:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:13:02.056434 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220019d2-2549-40fd-98f6-6f0e484c0248 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:13:02.058154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-220019d2-2549-40fd-98f6-6f0e484c0248 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:13:02.135831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220019d2-2549-40fd-98f6-6f0e484c0248 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:13:02.139453 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4d3ea8-cfcc-4c7b-b69e-1767ee7bb476 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:13:02.140053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4d3ea8-cfcc-4c7b-b69e-1767ee7bb476 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:13:02.140357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4d3ea8-cfcc-4c7b-b69e-1767ee7bb476 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:13:02.140636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4d3ea8-cfcc-4c7b-b69e-1767ee7bb476 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:13:02.149446 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4d3ea8-cfcc-4c7b-b69e-1767ee7bb476 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:13:02.150870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4d3ea8-cfcc-4c7b-b69e-1767ee7bb476 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:13:02.160621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 757/1530] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:13:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:13:04.322183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb75ab6-0073-4290-998c-bf5e72e51b39 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:13:04.322737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb75ab6-0073-4290-998c-bf5e72e51b39 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:13:04.322737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb75ab6-0073-4290-998c-bf5e72e51b39 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:13:04.324075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 774/1531] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:13:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:13:04.329423 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e40dba5-349e-4c78-9f68-1f1ae1287f4d None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:13:04.329837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e40dba5-349e-4c78-9f68-1f1ae1287f4d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:13:04.329984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e40dba5-349e-4c78-9f68-1f1ae1287f4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:13:04.350478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-0e40dba5-349e-4c78-9f68-1f1ae1287f4d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:13:04.351704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0e40dba5-349e-4c78-9f68-1f1ae1287f4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:13:04.360143 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 758/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:13:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:13:04.644400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb75ab6-0073-4290-998c-bf5e72e51b39 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:13:04.646097 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7eb75ab6-0073-4290-998c-bf5e72e51b39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:13:04.653980 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=75 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:13:04.661063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb75ab6-0073-4290-998c-bf5e72e51b39 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:13:04.664549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c97363-7b7e-4578-bd60-17a73746056c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:13:04.665134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c97363-7b7e-4578-bd60-17a73746056c service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:13:04.665413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c97363-7b7e-4578-bd60-17a73746056c service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:13:04.665684 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c97363-7b7e-4578-bd60-17a73746056c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:13:04.674597 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c97363-7b7e-4578-bd60-17a73746056c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:13:04.676020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c97363-7b7e-4578-bd60-17a73746056c service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:13:04.685378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 775/1533] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:13:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:13:08.527561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9342661-5dc0-478f-bea8-191cfb6f89f4 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:13:08.528105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9342661-5dc0-478f-bea8-191cfb6f89f4 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:13:08.528105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9342661-5dc0-478f-bea8-191cfb6f89f4 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:13:08.529489 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 759/1534] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:13:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:13:08.535536 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b3437e-6b9a-4bfe-b0a3-fb02852f2922 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:13:08.535968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b3437e-6b9a-4bfe-b0a3-fb02852f2922 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:13:08.536077 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b3437e-6b9a-4bfe-b0a3-fb02852f2922 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:13:08.555120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-38b3437e-6b9a-4bfe-b0a3-fb02852f2922 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:13:08.556330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-38b3437e-6b9a-4bfe-b0a3-fb02852f2922 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:13:08.564023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 776/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:13:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:13:08.865980 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9342661-5dc0-478f-bea8-191cfb6f89f4 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:13:08.867697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d9342661-5dc0-478f-bea8-191cfb6f89f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:13:08.931785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:13:08.945269 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9342661-5dc0-478f-bea8-191cfb6f89f4 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:13:08.948879 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4bd1f3-ecb1-42fd-8a86-4bd61ce14557 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:13:08.949477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4bd1f3-ecb1-42fd-8a86-4bd61ce14557 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:13:08.949759 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4bd1f3-ecb1-42fd-8a86-4bd61ce14557 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:13:08.950031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4bd1f3-ecb1-42fd-8a86-4bd61ce14557 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:13:08.958881 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4bd1f3-ecb1-42fd-8a86-4bd61ce14557 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:13:08.960339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4bd1f3-ecb1-42fd-8a86-4bd61ce14557 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:13:08.970501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 760/1536] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:13:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:13:11.111969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16eda86-ad56-44a8-82ad-38c9f2050c32 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:13:11.112543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16eda86-ad56-44a8-82ad-38c9f2050c32 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:13:11.112678 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16eda86-ad56-44a8-82ad-38c9f2050c32 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:13:11.113942 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 777/1537] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 28 12:13:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:13:11.119223 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092ca178-4fb2-4722-a38c-2a883633c23f None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:13:11.119626 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092ca178-4fb2-4722-a38c-2a883633c23f None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:13:11.119798 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092ca178-4fb2-4722-a38c-2a883633c23f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:13:11.139439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-092ca178-4fb2-4722-a38c-2a883633c23f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:13:11.141284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-092ca178-4fb2-4722-a38c-2a883633c23f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:13:11.149084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 761/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 28 12:13:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:13:11.446170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16eda86-ad56-44a8-82ad-38c9f2050c32 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:13:11.447799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a16eda86-ad56-44a8-82ad-38c9f2050c32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:13:11.461327 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16eda86-ad56-44a8-82ad-38c9f2050c32 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:13:11.464775 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856aaeab-6460-4e6e-bfd1-30c0fa57d921 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:13:11.465353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856aaeab-6460-4e6e-bfd1-30c0fa57d921 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:13:11.465627 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856aaeab-6460-4e6e-bfd1-30c0fa57d921 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:13:11.465893 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856aaeab-6460-4e6e-bfd1-30c0fa57d921 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:13:11.475553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856aaeab-6460-4e6e-bfd1-30c0fa57d921 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:13:11.476996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856aaeab-6460-4e6e-bfd1-30c0fa57d921 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:13:11.486165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 778/1539] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:13:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:13:21.134523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=11 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:13:21.482100 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=14 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:35.018965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bfc97c-68dd-4940-8c8b-45a01dfedca7 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.019596 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bfc97c-68dd-4940-8c8b-45a01dfedca7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.019596 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bfc97c-68dd-4940-8c8b-45a01dfedca7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.047392 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-71bfc97c-68dd-4940-8c8b-45a01dfedca7 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.048807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-71bfc97c-68dd-4940-8c8b-45a01dfedca7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.062346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 762/1540] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:14:35.071336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21db736f-f391-4a34-8ca9-108b15dad57e None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.072648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21db736f-f391-4a34-8ca9-108b15dad57e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.073157 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21db736f-f391-4a34-8ca9-108b15dad57e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.098537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e0b003-739c-4691-9851-74b43b9c5801 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.098894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e0b003-739c-4691-9851-74b43b9c5801 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.099096 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e0b003-739c-4691-9851-74b43b9c5801 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.109844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-21db736f-f391-4a34-8ca9-108b15dad57e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.111063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-21db736f-f391-4a34-8ca9-108b15dad57e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.123226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 779/1541] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:35.126569 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-18e0b003-739c-4691-9851-74b43b9c5801 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.127792 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-18e0b003-739c-4691-9851-74b43b9c5801 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.137032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21db736f-f391-4a34-8ca9-108b15dad57e None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:35.141518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 763/1542] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:35.149794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22260951-ba7c-446a-8156-2f0fbc04885d None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.150084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22260951-ba7c-446a-8156-2f0fbc04885d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.150361 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22260951-ba7c-446a-8156-2f0fbc04885d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.154831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21db736f-f391-4a34-8ca9-108b15dad57e None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:35.159636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8dcf12-5d8c-4acc-88e1-c2291b373a80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:35.160255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8dcf12-5d8c-4acc-88e1-c2291b373a80 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.160569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8dcf12-5d8c-4acc-88e1-c2291b373a80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.160845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8dcf12-5d8c-4acc-88e1-c2291b373a80 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.161503 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8dcf12-5d8c-4acc-88e1-c2291b373a80 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:14:35.161810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8dcf12-5d8c-4acc-88e1-c2291b373a80 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:35.162998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: WARNING py.warnings [None req-eb8dcf12-5d8c-4acc-88e1-c2291b373a80 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:14:35.162998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: warnings.warn(msg) Sep 28 12:14:35.162998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: Sep 28 12:14:35.163892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8dcf12-5d8c-4acc-88e1-c2291b373a80 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:35.179563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-22260951-ba7c-446a-8156-2f0fbc04885d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.180817 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-22260951-ba7c-446a-8156-2f0fbc04885d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.189394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 780/1543] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:14:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:14:35.201628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 764/1544] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:14:35.204282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8dcf12-5d8c-4acc-88e1-c2291b373a80 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:35.206927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22260951-ba7c-446a-8156-2f0fbc04885d None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:35.217823 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8dcf12-5d8c-4acc-88e1-c2291b373a80 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:35.225507 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22260951-ba7c-446a-8156-2f0fbc04885d None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:35.228101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e633e604-011d-4b20-9670-1191a9fc84ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:35.228798 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e633e604-011d-4b20-9670-1191a9fc84ae admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.229083 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e633e604-011d-4b20-9670-1191a9fc84ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.229359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e633e604-011d-4b20-9670-1191a9fc84ae admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.230129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e633e604-011d-4b20-9670-1191a9fc84ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:35.232191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e633e604-011d-4b20-9670-1191a9fc84ae admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:35.235062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef04a53c-c0e6-4d5c-92e2-722cb29a5208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:35.235630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef04a53c-c0e6-4d5c-92e2-722cb29a5208 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.235912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef04a53c-c0e6-4d5c-92e2-722cb29a5208 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.236187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef04a53c-c0e6-4d5c-92e2-722cb29a5208 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.236881 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef04a53c-c0e6-4d5c-92e2-722cb29a5208 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:14:35.237188 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef04a53c-c0e6-4d5c-92e2-722cb29a5208 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:35.238849 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef04a53c-c0e6-4d5c-92e2-722cb29a5208 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:35.261955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 765/1545] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:14:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:14:35.263382 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692a26b7-f5c8-4e8e-815a-2895e0e5d41d None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.263672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692a26b7-f5c8-4e8e-815a-2895e0e5d41d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.263946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692a26b7-f5c8-4e8e-815a-2895e0e5d41d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.301220 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-692a26b7-f5c8-4e8e-815a-2895e0e5d41d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.302596 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-692a26b7-f5c8-4e8e-815a-2895e0e5d41d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.319202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 781/1546] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/projects => generated 393 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:35.320039 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e633e604-011d-4b20-9670-1191a9fc84ae admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:35.339334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 766/1547] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:35.340220 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692a26b7-f5c8-4e8e-815a-2895e0e5d41d None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:35.366887 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692a26b7-f5c8-4e8e-815a-2895e0e5d41d None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:35.460087 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64392fe0-9c7f-48bb-8e94-e94a1280be1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:35.460742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64392fe0-9c7f-48bb-8e94-e94a1280be1f admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.461021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64392fe0-9c7f-48bb-8e94-e94a1280be1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.461295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64392fe0-9c7f-48bb-8e94-e94a1280be1f admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.461930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64392fe0-9c7f-48bb-8e94-e94a1280be1f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:35.463878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64392fe0-9c7f-48bb-8e94-e94a1280be1f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:35.471939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e633e604-011d-4b20-9670-1191a9fc84ae admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:35.496139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb840c9-ae76-402a-bf58-c51daf382952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:35.496920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb840c9-ae76-402a-bf58-c51daf382952 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.497321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb840c9-ae76-402a-bf58-c51daf382952 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.497604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb840c9-ae76-402a-bf58-c51daf382952 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.498270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb840c9-ae76-402a-bf58-c51daf382952 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:35.501925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb840c9-ae76-402a-bf58-c51daf382952 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:35.527033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 767/1548] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/users => generated 297 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:35.531907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40563268-d27f-4525-b4d7-26f9019ccfcb None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.531907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40563268-d27f-4525-b4d7-26f9019ccfcb None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.532133 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40563268-d27f-4525-b4d7-26f9019ccfcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.557548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 782/1549] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/projects => generated 377 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:35.559105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ab126f-c88f-4b9e-91ef-08f0705856e1 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.559494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ab126f-c88f-4b9e-91ef-08f0705856e1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.562078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-40563268-d27f-4525-b4d7-26f9019ccfcb None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.563304 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-40563268-d27f-4525-b4d7-26f9019ccfcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.565223 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ab126f-c88f-4b9e-91ef-08f0705856e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.586562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-96ab126f-c88f-4b9e-91ef-08f0705856e1 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.587758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-96ab126f-c88f-4b9e-91ef-08f0705856e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.625968 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 768/1550] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:35.627328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c27a3a-335b-4a9b-8d65-f9e83532f0ca None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.627616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c27a3a-335b-4a9b-8d65-f9e83532f0ca None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.627903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c27a3a-335b-4a9b-8d65-f9e83532f0ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.646363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-c5c27a3a-335b-4a9b-8d65-f9e83532f0ca None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.647553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c5c27a3a-335b-4a9b-8d65-f9e83532f0ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.650376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 783/1551] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:35.651614 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4a2bd9-0b6a-41d6-9d65-9308fe12f5b3 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.651900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4a2bd9-0b6a-41d6-9d65-9308fe12f5b3 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.652175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4a2bd9-0b6a-41d6-9d65-9308fe12f5b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.660629 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 769/1552] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:35.661679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c27a3a-335b-4a9b-8d65-f9e83532f0ca None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:35.672770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-ba4a2bd9-0b6a-41d6-9d65-9308fe12f5b3 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.673990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ba4a2bd9-0b6a-41d6-9d65-9308fe12f5b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.678605 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c27a3a-335b-4a9b-8d65-f9e83532f0ca None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:35.683081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 784/1553] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:35.684381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485e3c3a-5beb-464b-bd01-3ef5b0000b77 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.684702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485e3c3a-5beb-464b-bd01-3ef5b0000b77 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.684931 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485e3c3a-5beb-464b-bd01-3ef5b0000b77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.686237 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eae5e97-7f28-4784-8817-e4242484f19d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:35.687721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eae5e97-7f28-4784-8817-e4242484f19d admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.688384 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eae5e97-7f28-4784-8817-e4242484f19d admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.688834 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eae5e97-7f28-4784-8817-e4242484f19d admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.689702 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eae5e97-7f28-4784-8817-e4242484f19d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:35.692803 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eae5e97-7f28-4784-8817-e4242484f19d admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:35.704211 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 770/1554] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:35 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:35.705012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eae5e97-7f28-4784-8817-e4242484f19d admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:35.706574 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-485e3c3a-5beb-464b-bd01-3ef5b0000b77 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.707948 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-485e3c3a-5beb-464b-bd01-3ef5b0000b77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.714645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eae5e97-7f28-4784-8817-e4242484f19d admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:35.720413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 785/1555] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:35.720413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247640ec-3692-4414-b954-33b8efa2034c None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.720413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247640ec-3692-4414-b954-33b8efa2034c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.720413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247640ec-3692-4414-b954-33b8efa2034c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.722345 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1987cda0-549a-426f-ba26-6290b1b07d68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:35.722929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1987cda0-549a-426f-ba26-6290b1b07d68 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.723208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1987cda0-549a-426f-ba26-6290b1b07d68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.723476 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1987cda0-549a-426f-ba26-6290b1b07d68 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.724387 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1987cda0-549a-426f-ba26-6290b1b07d68 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:35.726354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1987cda0-549a-426f-ba26-6290b1b07d68 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:35.738838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-247640ec-3692-4414-b954-33b8efa2034c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.740097 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-247640ec-3692-4414-b954-33b8efa2034c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.749399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 786/1556] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:35.749399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247640ec-3692-4414-b954-33b8efa2034c None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:35.751073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 771/1557] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/users => generated 289 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:35.755873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb7557b-13c2-4916-b06e-d96f30753282 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.755873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb7557b-13c2-4916-b06e-d96f30753282 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.755873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb7557b-13c2-4916-b06e-d96f30753282 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.761760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247640ec-3692-4414-b954-33b8efa2034c None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:35.767003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf370b2-232f-4a8a-a101-4ce66c3cccb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:35.767599 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf370b2-232f-4a8a-a101-4ce66c3cccb1 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.767878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf370b2-232f-4a8a-a101-4ce66c3cccb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.768155 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf370b2-232f-4a8a-a101-4ce66c3cccb1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.768837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf370b2-232f-4a8a-a101-4ce66c3cccb1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:14:35.769136 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf370b2-232f-4a8a-a101-4ce66c3cccb1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:35.770721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf370b2-232f-4a8a-a101-4ce66c3cccb1 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:35.773969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-dfb7557b-13c2-4916-b06e-d96f30753282 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.775180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-dfb7557b-13c2-4916-b06e-d96f30753282 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.789243 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 787/1558] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:14:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:14:35.789431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 772/1559] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:35.789560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537dfee2-c73c-4c9a-b2c6-95c5974912ff None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.789560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537dfee2-c73c-4c9a-b2c6-95c5974912ff None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.789560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537dfee2-c73c-4c9a-b2c6-95c5974912ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.789835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cebaca5-1f3d-41fb-b6fa-a6858ba4e65f None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.789835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cebaca5-1f3d-41fb-b6fa-a6858ba4e65f None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.789835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cebaca5-1f3d-41fb-b6fa-a6858ba4e65f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.794696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:35.807410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-537dfee2-c73c-4c9a-b2c6-95c5974912ff None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.808645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-537dfee2-c73c-4c9a-b2c6-95c5974912ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.810099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-9cebaca5-1f3d-41fb-b6fa-a6858ba4e65f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.811250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9cebaca5-1f3d-41fb-b6fa-a6858ba4e65f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.817123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 788/1560] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:35.817969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537dfee2-c73c-4c9a-b2c6-95c5974912ff None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:35.819668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 773/1561] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:35.820864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bada84c-8e87-411f-8f10-119205106951 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.821268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bada84c-8e87-411f-8f10-119205106951 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.821544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bada84c-8e87-411f-8f10-119205106951 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.829675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:35.832361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537dfee2-c73c-4c9a-b2c6-95c5974912ff None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:35.838082 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a9152f-d151-445b-89f5-223ea771a0b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:35.839253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a9152f-d151-445b-89f5-223ea771a0b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.839720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a9152f-d151-445b-89f5-223ea771a0b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.840137 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a9152f-d151-445b-89f5-223ea771a0b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.841347 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a9152f-d151-445b-89f5-223ea771a0b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:35.843088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a9152f-d151-445b-89f5-223ea771a0b8 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:35.843922 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-4bada84c-8e87-411f-8f10-119205106951 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.845534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-4bada84c-8e87-411f-8f10-119205106951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.853757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 789/1562] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:35 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:35.854547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a9152f-d151-445b-89f5-223ea771a0b8 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:35.859811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 774/1563] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:35.859811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d602ad0c-9910-46ca-a07e-4db47c6538cb None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.859811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d602ad0c-9910-46ca-a07e-4db47c6538cb None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.859811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d602ad0c-9910-46ca-a07e-4db47c6538cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.864714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a9152f-d151-445b-89f5-223ea771a0b8 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:35.869295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21dc5c6-5cc0-452d-970f-91649227ae69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:35.869882 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21dc5c6-5cc0-452d-970f-91649227ae69 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.870159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21dc5c6-5cc0-452d-970f-91649227ae69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.870431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21dc5c6-5cc0-452d-970f-91649227ae69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.871071 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21dc5c6-5cc0-452d-970f-91649227ae69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:35.872666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21dc5c6-5cc0-452d-970f-91649227ae69 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:35.880010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-d602ad0c-9910-46ca-a07e-4db47c6538cb None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:35.881235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d602ad0c-9910-46ca-a07e-4db47c6538cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:35.883217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 790/1564] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:35 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:35.885185 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21dc5c6-5cc0-452d-970f-91649227ae69 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:35.889370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 775/1565] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:35.890305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d602ad0c-9910-46ca-a07e-4db47c6538cb None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:35.897701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21dc5c6-5cc0-452d-970f-91649227ae69 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:35.899520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d602ad0c-9910-46ca-a07e-4db47c6538cb None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:35.902366 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fb8a8a-7696-4a21-8192-551d591a10bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:35.902947 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fb8a8a-7696-4a21-8192-551d591a10bd admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.903226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fb8a8a-7696-4a21-8192-551d591a10bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.903494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fb8a8a-7696-4a21-8192-551d591a10bd admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.904051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad03ed2-dc72-4ca4-b1c2-e07d824a71c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:35.904188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fb8a8a-7696-4a21-8192-551d591a10bd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:14:35.904449 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fb8a8a-7696-4a21-8192-551d591a10bd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:35.904663 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad03ed2-dc72-4ca4-b1c2-e07d824a71c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.904942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad03ed2-dc72-4ca4-b1c2-e07d824a71c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.905240 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad03ed2-dc72-4ca4-b1c2-e07d824a71c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.905862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad03ed2-dc72-4ca4-b1c2-e07d824a71c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:35.906027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fb8a8a-7696-4a21-8192-551d591a10bd admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:35.906871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: WARNING py.warnings [None req-fad03ed2-dc72-4ca4-b1c2-e07d824a71c8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:14:35.906871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: warnings.warn(msg) Sep 28 12:14:35.906871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: Sep 28 12:14:35.908048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad03ed2-dc72-4ca4-b1c2-e07d824a71c8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:35.918060 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 791/1566] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:14:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:14:35.924126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fb8a8a-7696-4a21-8192-551d591a10bd admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:35.935955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fb8a8a-7696-4a21-8192-551d591a10bd admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:35.941343 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 776/1567] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:35 2022] POST /identity/v3/projects => generated 383 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:35.941343 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad03ed2-dc72-4ca4-b1c2-e07d824a71c8 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:35.992843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44dc69d-108d-4dfd-aba7-2b87e19b915b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:35.993397 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44dc69d-108d-4dfd-aba7-2b87e19b915b admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:35.993678 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44dc69d-108d-4dfd-aba7-2b87e19b915b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:35.993950 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44dc69d-108d-4dfd-aba7-2b87e19b915b admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:35.994649 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44dc69d-108d-4dfd-aba7-2b87e19b915b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:14:35.994883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44dc69d-108d-4dfd-aba7-2b87e19b915b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:35.996479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44dc69d-108d-4dfd-aba7-2b87e19b915b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.004358 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad03ed2-dc72-4ca4-b1c2-e07d824a71c8 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.008055 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 792/1568] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:14:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:14:36.008874 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44dc69d-108d-4dfd-aba7-2b87e19b915b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.009029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c14bc4-b560-4ed6-8747-d1fdb986c86a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.009632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c14bc4-b560-4ed6-8747-d1fdb986c86a admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.009919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c14bc4-b560-4ed6-8747-d1fdb986c86a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.010194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c14bc4-b560-4ed6-8747-d1fdb986c86a admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.010840 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c14bc4-b560-4ed6-8747-d1fdb986c86a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:14:36.011144 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c14bc4-b560-4ed6-8747-d1fdb986c86a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.012790 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c14bc4-b560-4ed6-8747-d1fdb986c86a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.020368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44dc69d-108d-4dfd-aba7-2b87e19b915b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.024779 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 777/1569] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:14:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:14:36.032539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b53143-4dbb-4bb3-b3dc-742da1a28487 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.032666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c14bc4-b560-4ed6-8747-d1fdb986c86a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.032772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b53143-4dbb-4bb3-b3dc-742da1a28487 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.032772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b53143-4dbb-4bb3-b3dc-742da1a28487 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.032772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b53143-4dbb-4bb3-b3dc-742da1a28487 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.032772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b53143-4dbb-4bb3-b3dc-742da1a28487 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:14:36.032772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b53143-4dbb-4bb3-b3dc-742da1a28487 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.032772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b53143-4dbb-4bb3-b3dc-742da1a28487 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.047603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 793/1570] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:14:36.047787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c14bc4-b560-4ed6-8747-d1fdb986c86a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.047903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b53143-4dbb-4bb3-b3dc-742da1a28487 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.048007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512919a8-9267-4f90-8766-a4c83ede3268 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.048007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512919a8-9267-4f90-8766-a4c83ede3268 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.048007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512919a8-9267-4f90-8766-a4c83ede3268 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.048007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512919a8-9267-4f90-8766-a4c83ede3268 admin admin] PATH_INFO: `/v3/projects/47157b4286ac4ed5a24ba31bafd206a2/users/956c7f92a4a747dba49e3f3a527bba74/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.056998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b53143-4dbb-4bb3-b3dc-742da1a28487 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.062292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f625196d-fcb5-4541-beaf-4c70c8eaa564 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.062861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512919a8-9267-4f90-8766-a4c83ede3268 admin admin] RBAC: Authorizing `identity:create_grant(project_id=47157b4286ac4ed5a24ba31bafd206a2, user_id=956c7f92a4a747dba49e3f3a527bba74, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.063001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f625196d-fcb5-4541-beaf-4c70c8eaa564 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.063138 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f625196d-fcb5-4541-beaf-4c70c8eaa564 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.063406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f625196d-fcb5-4541-beaf-4c70c8eaa564 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.063897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: WARNING py.warnings [None req-512919a8-9267-4f90-8766-a4c83ede3268 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:14:36.063897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: warnings.warn(msg) Sep 28 12:14:36.063897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: Sep 28 12:14:36.064156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f625196d-fcb5-4541-beaf-4c70c8eaa564 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.065611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f625196d-fcb5-4541-beaf-4c70c8eaa564 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.065865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512919a8-9267-4f90-8766-a4c83ede3268 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.076121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 794/1571] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:36.076930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f625196d-fcb5-4541-beaf-4c70c8eaa564 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.081757 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 778/1572] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:36 2022] PUT /identity/v3/projects/47157b4286ac4ed5a24ba31bafd206a2/users/956c7f92a4a747dba49e3f3a527bba74/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:36.082529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512919a8-9267-4f90-8766-a4c83ede3268 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.087517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f625196d-fcb5-4541-beaf-4c70c8eaa564 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.144570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512919a8-9267-4f90-8766-a4c83ede3268 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.144958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab3a39d-c6ad-49ad-bd88-f0c12efb2033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.145568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab3a39d-c6ad-49ad-bd88-f0c12efb2033 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.145848 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab3a39d-c6ad-49ad-bd88-f0c12efb2033 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.146121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab3a39d-c6ad-49ad-bd88-f0c12efb2033 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.146768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab3a39d-c6ad-49ad-bd88-f0c12efb2033 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:14:36.147071 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab3a39d-c6ad-49ad-bd88-f0c12efb2033 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.148757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab3a39d-c6ad-49ad-bd88-f0c12efb2033 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.149156 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3667ba76-c6f3-4668-83e2-0d86b28eacc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.149748 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3667ba76-c6f3-4668-83e2-0d86b28eacc8 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.150026 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3667ba76-c6f3-4668-83e2-0d86b28eacc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.150342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3667ba76-c6f3-4668-83e2-0d86b28eacc8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.150969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3667ba76-c6f3-4668-83e2-0d86b28eacc8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.152899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3667ba76-c6f3-4668-83e2-0d86b28eacc8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.160971 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 795/1573] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:14:36.161910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab3a39d-c6ad-49ad-bd88-f0c12efb2033 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.172083 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab3a39d-c6ad-49ad-bd88-f0c12efb2033 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.177423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13365a6-4051-4cfd-9a31-4bdcb38b47c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.178125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13365a6-4051-4cfd-9a31-4bdcb38b47c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.178399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13365a6-4051-4cfd-9a31-4bdcb38b47c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.178663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13365a6-4051-4cfd-9a31-4bdcb38b47c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.179281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d13365a6-4051-4cfd-9a31-4bdcb38b47c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.180266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: WARNING py.warnings [None req-d13365a6-4051-4cfd-9a31-4bdcb38b47c6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:14:36.180266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: warnings.warn(msg) Sep 28 12:14:36.180266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: Sep 28 12:14:36.181475 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d13365a6-4051-4cfd-9a31-4bdcb38b47c6 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.184350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 779/1574] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:36 2022] POST /identity/v3/projects => generated 365 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:36.185173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3667ba76-c6f3-4668-83e2-0d86b28eacc8 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.204700 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 796/1575] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:36 2022] POST /identity/v3/users => generated 292 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:36.205494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13365a6-4051-4cfd-9a31-4bdcb38b47c6 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.215015 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13365a6-4051-4cfd-9a31-4bdcb38b47c6 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.215390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:36.219253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:36.253740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3667ba76-c6f3-4668-83e2-0d86b28eacc8 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.258570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665a4bbf-9d36-450b-9367-22c5447f0e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.259101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665a4bbf-9d36-450b-9367-22c5447f0e75 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.259381 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665a4bbf-9d36-450b-9367-22c5447f0e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.259778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665a4bbf-9d36-450b-9367-22c5447f0e75 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.260477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665a4bbf-9d36-450b-9367-22c5447f0e75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.262431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665a4bbf-9d36-450b-9367-22c5447f0e75 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.274058 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d1a8de-92f2-4e4e-ae0e-390b0424be31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.274715 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d1a8de-92f2-4e4e-ae0e-390b0424be31 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.275002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d1a8de-92f2-4e4e-ae0e-390b0424be31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.275274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d1a8de-92f2-4e4e-ae0e-390b0424be31 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.275907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d1a8de-92f2-4e4e-ae0e-390b0424be31 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.277811 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d1a8de-92f2-4e4e-ae0e-390b0424be31 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.290420 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 780/1576] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:36 2022] POST /identity/v3/projects => generated 403 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:36.291224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665a4bbf-9d36-450b-9367-22c5447f0e75 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.308019 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 797/1577] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:36 2022] POST /identity/v3/projects => generated 387 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:36.308996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d1a8de-92f2-4e4e-ae0e-390b0424be31 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.356209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665a4bbf-9d36-450b-9367-22c5447f0e75 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.369991 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d1a8de-92f2-4e4e-ae0e-390b0424be31 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.375295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4737d7-8968-49d5-b127-c3d038f7d2f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.375950 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4737d7-8968-49d5-b127-c3d038f7d2f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.376233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4737d7-8968-49d5-b127-c3d038f7d2f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.376539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4737d7-8968-49d5-b127-c3d038f7d2f8 admin admin] PATH_INFO: `/v3/projects/f7f6e096c5a54a7ebc843b012e82e0f4/users/fdb0f345d2e04483b5166e1eb728bf91/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.391248 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4737d7-8968-49d5-b127-c3d038f7d2f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7f6e096c5a54a7ebc843b012e82e0f4, user_id=fdb0f345d2e04483b5166e1eb728bf91, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.393939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4737d7-8968-49d5-b127-c3d038f7d2f8 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.407406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 798/1578] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:36 2022] PUT /identity/v3/projects/f7f6e096c5a54a7ebc843b012e82e0f4/users/fdb0f345d2e04483b5166e1eb728bf91/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:36.408689 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f31121f-4f3e-4379-9b1e-a372896638c8 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.408970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f31121f-4f3e-4379-9b1e-a372896638c8 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.409236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f31121f-4f3e-4379-9b1e-a372896638c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.412866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83f2114-16c6-4f86-9767-55c764c31eae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.413506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83f2114-16c6-4f86-9767-55c764c31eae admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.413813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83f2114-16c6-4f86-9767-55c764c31eae admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.414083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83f2114-16c6-4f86-9767-55c764c31eae admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.414714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83f2114-16c6-4f86-9767-55c764c31eae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.416787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83f2114-16c6-4f86-9767-55c764c31eae admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.427229 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4f31121f-4f3e-4379-9b1e-a372896638c8 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:36.428475 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4f31121f-4f3e-4379-9b1e-a372896638c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:36.443853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 781/1579] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:36 2022] POST /identity/v3/projects => generated 355 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:36.447812 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83f2114-16c6-4f86-9767-55c764c31eae admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.492070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 799/1580] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:36.492070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f31121f-4f3e-4379-9b1e-a372896638c8 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.500184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f31121f-4f3e-4379-9b1e-a372896638c8 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.500730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:36.512800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:36.522265 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83f2114-16c6-4f86-9767-55c764c31eae admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.526912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491262f0-0a3b-4cc8-a756-9e8c44a90920 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.527513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491262f0-0a3b-4cc8-a756-9e8c44a90920 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.527790 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491262f0-0a3b-4cc8-a756-9e8c44a90920 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.528056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491262f0-0a3b-4cc8-a756-9e8c44a90920 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.528710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491262f0-0a3b-4cc8-a756-9e8c44a90920 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.530678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491262f0-0a3b-4cc8-a756-9e8c44a90920 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.558514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 782/1581] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:36 2022] POST /identity/v3/projects => generated 375 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:36.559321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491262f0-0a3b-4cc8-a756-9e8c44a90920 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.563804 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9fecf3-ca2d-48cd-95dd-dca5435b0a29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.564849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9fecf3-ca2d-48cd-95dd-dca5435b0a29 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.565214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9fecf3-ca2d-48cd-95dd-dca5435b0a29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.565486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9fecf3-ca2d-48cd-95dd-dca5435b0a29 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.566140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9fecf3-ca2d-48cd-95dd-dca5435b0a29 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.569304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9fecf3-ca2d-48cd-95dd-dca5435b0a29 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.591579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 800/1582] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:36 2022] POST /identity/v3/users => generated 283 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:36.592962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9fecf3-ca2d-48cd-95dd-dca5435b0a29 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.603565 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9fecf3-ca2d-48cd-95dd-dca5435b0a29 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.623202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491262f0-0a3b-4cc8-a756-9e8c44a90920 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.628383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e497653-6f16-4d91-9913-4a7fd12c4303 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.628981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e497653-6f16-4d91-9913-4a7fd12c4303 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.629257 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e497653-6f16-4d91-9913-4a7fd12c4303 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.629527 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e497653-6f16-4d91-9913-4a7fd12c4303 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.630153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e497653-6f16-4d91-9913-4a7fd12c4303 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.631739 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e497653-6f16-4d91-9913-4a7fd12c4303 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.645171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 783/1583] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:36.649113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e497653-6f16-4d91-9913-4a7fd12c4303 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.658403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e497653-6f16-4d91-9913-4a7fd12c4303 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.658679 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d42085-1394-43e3-b816-b160b1a065b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.659285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d42085-1394-43e3-b816-b160b1a065b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.659560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d42085-1394-43e3-b816-b160b1a065b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.659830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d42085-1394-43e3-b816-b160b1a065b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.660482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d42085-1394-43e3-b816-b160b1a065b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.662625 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d42085-1394-43e3-b816-b160b1a065b2 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.663696 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80663be-d447-44ba-b179-effe7da4b536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.664301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80663be-d447-44ba-b179-effe7da4b536 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.664636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80663be-d447-44ba-b179-effe7da4b536 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.664912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80663be-d447-44ba-b179-effe7da4b536 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.665533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80663be-d447-44ba-b179-effe7da4b536 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.667431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80663be-d447-44ba-b179-effe7da4b536 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.685445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 801/1584] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:36 2022] POST /identity/v3/users => generated 302 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:36.687303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd3425b-3585-4ef2-9dc2-378441282d7d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.687590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd3425b-3585-4ef2-9dc2-378441282d7d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.687861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd3425b-3585-4ef2-9dc2-378441282d7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.695326 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 784/1585] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:36 2022] POST /identity/v3/users => generated 294 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:36.695326 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80663be-d447-44ba-b179-effe7da4b536 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.703827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80663be-d447-44ba-b179-effe7da4b536 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.707313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-ebd3425b-3585-4ef2-9dc2-378441282d7d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:36.708536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f34f8a-0033-4320-960d-84f2e07b471c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.708689 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ebd3425b-3585-4ef2-9dc2-378441282d7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:36.709120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f34f8a-0033-4320-960d-84f2e07b471c admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.709399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f34f8a-0033-4320-960d-84f2e07b471c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.709670 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f34f8a-0033-4320-960d-84f2e07b471c admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.710288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f34f8a-0033-4320-960d-84f2e07b471c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.712194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f34f8a-0033-4320-960d-84f2e07b471c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.716204 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 802/1586] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:36.717004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd3425b-3585-4ef2-9dc2-378441282d7d None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.740933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 785/1587] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:36 2022] POST /identity/v3/users => generated 278 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:36.740933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f34f8a-0033-4320-960d-84f2e07b471c admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.748422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f34f8a-0033-4320-960d-84f2e07b471c admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.753608 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e09d57-31a7-4bf5-8d12-cea3a5045bc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.754262 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e09d57-31a7-4bf5-8d12-cea3a5045bc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.754540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e09d57-31a7-4bf5-8d12-cea3a5045bc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.754851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e09d57-31a7-4bf5-8d12-cea3a5045bc5 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.755477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e09d57-31a7-4bf5-8d12-cea3a5045bc5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.757447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e09d57-31a7-4bf5-8d12-cea3a5045bc5 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.776147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd3425b-3585-4ef2-9dc2-378441282d7d None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.778942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 786/1588] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:36 2022] POST /identity/v3/users => generated 288 bytes in 40 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:36.779774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e09d57-31a7-4bf5-8d12-cea3a5045bc5 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.779889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71d5847-9521-46f6-87b7-c5e13ddef8f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.780501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71d5847-9521-46f6-87b7-c5e13ddef8f1 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.780892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71d5847-9521-46f6-87b7-c5e13ddef8f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.781173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71d5847-9521-46f6-87b7-c5e13ddef8f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.789868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e09d57-31a7-4bf5-8d12-cea3a5045bc5 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.790729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71d5847-9521-46f6-87b7-c5e13ddef8f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.792190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71d5847-9521-46f6-87b7-c5e13ddef8f1 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.794625 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d58ae7-0ac1-417d-9ddf-26fb8b5ad263 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.795177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d58ae7-0ac1-417d-9ddf-26fb8b5ad263 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.795461 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d58ae7-0ac1-417d-9ddf-26fb8b5ad263 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.796233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d58ae7-0ac1-417d-9ddf-26fb8b5ad263 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.797084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d58ae7-0ac1-417d-9ddf-26fb8b5ad263 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.798838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d58ae7-0ac1-417d-9ddf-26fb8b5ad263 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.802478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 803/1589] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:36.803379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71d5847-9521-46f6-87b7-c5e13ddef8f1 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.808633 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 787/1590] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:36.809505 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d58ae7-0ac1-417d-9ddf-26fb8b5ad263 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.813269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71d5847-9521-46f6-87b7-c5e13ddef8f1 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.818681 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa99a24-e390-41cf-a527-49d3fc806bdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.819883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa99a24-e390-41cf-a527-49d3fc806bdb admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.820170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa99a24-e390-41cf-a527-49d3fc806bdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.820482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa99a24-e390-41cf-a527-49d3fc806bdb admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.821129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa99a24-e390-41cf-a527-49d3fc806bdb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.822714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa99a24-e390-41cf-a527-49d3fc806bdb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.822981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d58ae7-0ac1-417d-9ddf-26fb8b5ad263 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.828574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8aae65-d9a5-4fa8-a2ee-378fe629fb29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.829166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8aae65-d9a5-4fa8-a2ee-378fe629fb29 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.829462 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8aae65-d9a5-4fa8-a2ee-378fe629fb29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.829700 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8aae65-d9a5-4fa8-a2ee-378fe629fb29 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.830341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8aae65-d9a5-4fa8-a2ee-378fe629fb29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.832473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8aae65-d9a5-4fa8-a2ee-378fe629fb29 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.834185 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 804/1591] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:36.835170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa99a24-e390-41cf-a527-49d3fc806bdb admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.845708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa99a24-e390-41cf-a527-49d3fc806bdb admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.849906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 788/1592] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:36.849906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8aae65-d9a5-4fa8-a2ee-378fe629fb29 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.854112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7719e692-5442-4bc2-b7c0-c71556e9a6a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.854750 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7719e692-5442-4bc2-b7c0-c71556e9a6a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.855032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7719e692-5442-4bc2-b7c0-c71556e9a6a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.855369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7719e692-5442-4bc2-b7c0-c71556e9a6a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.856015 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7719e692-5442-4bc2-b7c0-c71556e9a6a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.857762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7719e692-5442-4bc2-b7c0-c71556e9a6a7 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.860978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8aae65-d9a5-4fa8-a2ee-378fe629fb29 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.864801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8275fcd-b843-44a5-9767-a01643497b4f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.865457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8275fcd-b843-44a5-9767-a01643497b4f service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.865738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8275fcd-b843-44a5-9767-a01643497b4f service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.866033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8275fcd-b843-44a5-9767-a01643497b4f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.871704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 805/1593] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:36.871704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7719e692-5442-4bc2-b7c0-c71556e9a6a7 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.879391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8275fcd-b843-44a5-9767-a01643497b4f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.880778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7719e692-5442-4bc2-b7c0-c71556e9a6a7 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.881040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8275fcd-b843-44a5-9767-a01643497b4f service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.885720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61016279-74b4-4947-a3fc-3b2827772152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.886474 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61016279-74b4-4947-a3fc-3b2827772152 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.886857 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61016279-74b4-4947-a3fc-3b2827772152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.887199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61016279-74b4-4947-a3fc-3b2827772152 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.887952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61016279-74b4-4947-a3fc-3b2827772152 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.889907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61016279-74b4-4947-a3fc-3b2827772152 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.892634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 789/1594] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:36.893107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:36.893448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8275fcd-b843-44a5-9767-a01643497b4f service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.904805 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8275fcd-b843-44a5-9767-a01643497b4f service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.906590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 806/1595] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:36.907712 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61016279-74b4-4947-a3fc-3b2827772152 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.914546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a954df76-f521-4009-a2a7-50e1be1b8d2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.915184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a954df76-f521-4009-a2a7-50e1be1b8d2e admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.915590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a954df76-f521-4009-a2a7-50e1be1b8d2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.915759 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a954df76-f521-4009-a2a7-50e1be1b8d2e admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.916435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a954df76-f521-4009-a2a7-50e1be1b8d2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.918039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a954df76-f521-4009-a2a7-50e1be1b8d2e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.918391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61016279-74b4-4947-a3fc-3b2827772152 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.924020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddda558-7139-475e-8cf0-57bbf94af754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.924768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddda558-7139-475e-8cf0-57bbf94af754 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.925204 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddda558-7139-475e-8cf0-57bbf94af754 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.925609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddda558-7139-475e-8cf0-57bbf94af754 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.926411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddda558-7139-475e-8cf0-57bbf94af754 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.928225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddda558-7139-475e-8cf0-57bbf94af754 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.929059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 790/1596] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:36.929834 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a954df76-f521-4009-a2a7-50e1be1b8d2e admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.940001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a954df76-f521-4009-a2a7-50e1be1b8d2e admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.945460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 807/1597] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:36.946357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddda558-7139-475e-8cf0-57bbf94af754 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.950068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8485b6e4-9081-4d22-965a-d559c06f5f22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.950788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8485b6e4-9081-4d22-965a-d559c06f5f22 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.951058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8485b6e4-9081-4d22-965a-d559c06f5f22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.951485 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8485b6e4-9081-4d22-965a-d559c06f5f22 admin admin] PATH_INFO: `/v3/projects/81f514e581a3473188c0ca78416919a4/users/4a8818fd59954480bcff68b9ba52769a/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.959606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddda558-7139-475e-8cf0-57bbf94af754 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:36.967255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81adaf7d-f414-4b97-a57c-e106f238933e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:36.968848 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81adaf7d-f414-4b97-a57c-e106f238933e admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:36.969196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81adaf7d-f414-4b97-a57c-e106f238933e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:36.969541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81adaf7d-f414-4b97-a57c-e106f238933e admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:36.969980 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8485b6e4-9081-4d22-965a-d559c06f5f22 admin admin] RBAC: Authorizing `identity:create_grant(project_id=81f514e581a3473188c0ca78416919a4, user_id=4a8818fd59954480bcff68b9ba52769a, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.970333 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81adaf7d-f414-4b97-a57c-e106f238933e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:36.971985 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81adaf7d-f414-4b97-a57c-e106f238933e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.973000 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8485b6e4-9081-4d22-965a-d559c06f5f22 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:36.981831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:INSERT=11 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:36.987651 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 808/1598] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:36.987651 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81adaf7d-f414-4b97-a57c-e106f238933e admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.992585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 791/1599] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:36 2022] PUT /identity/v3/projects/81f514e581a3473188c0ca78416919a4/users/4a8818fd59954480bcff68b9ba52769a/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:36.994350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8485b6e4-9081-4d22-965a-d559c06f5f22 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:36.996958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81adaf7d-f414-4b97-a57c-e106f238933e admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:37.069735 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e579b276-7cba-4ea4-b45e-9e6d0c189e9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:37.070504 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e579b276-7cba-4ea4-b45e-9e6d0c189e9f admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.070970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e579b276-7cba-4ea4-b45e-9e6d0c189e9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.071314 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e579b276-7cba-4ea4-b45e-9e6d0c189e9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.072016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e579b276-7cba-4ea4-b45e-9e6d0c189e9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:37.073670 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e579b276-7cba-4ea4-b45e-9e6d0c189e9f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:37.075323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8485b6e4-9081-4d22-965a-d559c06f5f22 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:37.084224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7997f20e-5d31-4430-b1b5-b7ef97bd04a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:37.085107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7997f20e-5d31-4430-b1b5-b7ef97bd04a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.085549 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7997f20e-5d31-4430-b1b5-b7ef97bd04a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.085970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 809/1600] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:37.086571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7997f20e-5d31-4430-b1b5-b7ef97bd04a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.086856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e579b276-7cba-4ea4-b45e-9e6d0c189e9f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:37.087636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7997f20e-5d31-4430-b1b5-b7ef97bd04a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:37.090181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7997f20e-5d31-4430-b1b5-b7ef97bd04a2 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:37.097797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e579b276-7cba-4ea4-b45e-9e6d0c189e9f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:37.102564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a883840-221c-4988-b172-b225a4419a54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:37.103154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a883840-221c-4988-b172-b225a4419a54 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.103430 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a883840-221c-4988-b172-b225a4419a54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.103699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a883840-221c-4988-b172-b225a4419a54 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.104372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a883840-221c-4988-b172-b225a4419a54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:37.104537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 792/1601] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:36 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:37.105528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7997f20e-5d31-4430-b1b5-b7ef97bd04a2 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:37.105954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a883840-221c-4988-b172-b225a4419a54 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:37.120199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 810/1602] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:37 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:37.120199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a883840-221c-4988-b172-b225a4419a54 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:37.122089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7997f20e-5d31-4430-b1b5-b7ef97bd04a2 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:37.128264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a883840-221c-4988-b172-b225a4419a54 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:37.129167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cc8406-ee1f-439c-b280-07114e7d92e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:37.130130 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cc8406-ee1f-439c-b280-07114e7d92e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.130726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cc8406-ee1f-439c-b280-07114e7d92e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.130726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cc8406-ee1f-439c-b280-07114e7d92e4 admin admin] PATH_INFO: `/v3/projects/3992ec06d59148758b6883f90f0b4f06/users/fe401c67b1d54c2c86dd1e6f289f5b11/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.133277 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99efb6ac-a9e7-431f-9181-9f99730a021a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:37.133920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99efb6ac-a9e7-431f-9181-9f99730a021a admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.134195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99efb6ac-a9e7-431f-9181-9f99730a021a admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.135252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99efb6ac-a9e7-431f-9181-9f99730a021a admin admin] PATH_INFO: `/v3/projects/4755e00b2b02475ea04b589bb7865d92/users/4e6b3e1f654d4365a6b6ead804444986/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.150128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99efb6ac-a9e7-431f-9181-9f99730a021a admin admin] RBAC: Authorizing `identity:create_grant(project_id=4755e00b2b02475ea04b589bb7865d92, user_id=4e6b3e1f654d4365a6b6ead804444986, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:37.152562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cc8406-ee1f-439c-b280-07114e7d92e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3992ec06d59148758b6883f90f0b4f06, user_id=fe401c67b1d54c2c86dd1e6f289f5b11, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:37.152864 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99efb6ac-a9e7-431f-9181-9f99730a021a admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:37.156137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cc8406-ee1f-439c-b280-07114e7d92e4 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:37.169426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 811/1603] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:37 2022] PUT /identity/v3/projects/4755e00b2b02475ea04b589bb7865d92/users/4e6b3e1f654d4365a6b6ead804444986/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:37.170697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c86e7cb-8864-4893-9c57-068b82142e7d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.171063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c86e7cb-8864-4893-9c57-068b82142e7d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.171530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c86e7cb-8864-4893-9c57-068b82142e7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.176378 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 793/1604] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:37 2022] PUT /identity/v3/projects/3992ec06d59148758b6883f90f0b4f06/users/fe401c67b1d54c2c86dd1e6f289f5b11/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:37.178727 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cc8406-ee1f-439c-b280-07114e7d92e4 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:37.200644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-5c86e7cb-8864-4893-9c57-068b82142e7d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:37.201866 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5c86e7cb-8864-4893-9c57-068b82142e7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:37.294229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cc8406-ee1f-439c-b280-07114e7d92e4 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:37.297526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 812/1605] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:37.299877 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a9a1d1-71c9-46ce-9e26-4b6013d1560e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:37.300748 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a9a1d1-71c9-46ce-9e26-4b6013d1560e admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.301029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a9a1d1-71c9-46ce-9e26-4b6013d1560e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.301303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a9a1d1-71c9-46ce-9e26-4b6013d1560e admin admin] PATH_INFO: `/v3/projects/ba288a6bdc4e41729bd679733eaee61e/users/820f4bb963644499a77fa1f3edd59d08/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.301859 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c86e7cb-8864-4893-9c57-068b82142e7d None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:37.314451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c86e7cb-8864-4893-9c57-068b82142e7d None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:37.319900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c42e46-897c-4595-93c4-da6b845ab345 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:37.320692 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c42e46-897c-4595-93c4-da6b845ab345 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.321046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c42e46-897c-4595-93c4-da6b845ab345 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.321397 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c42e46-897c-4595-93c4-da6b845ab345 admin admin] PATH_INFO: `/v3/projects/66c178066d9e4b86835ed533fcd680cb/users/d90c463f98244312b78e998f4df89188/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.324069 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a9a1d1-71c9-46ce-9e26-4b6013d1560e admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba288a6bdc4e41729bd679733eaee61e, user_id=820f4bb963644499a77fa1f3edd59d08, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:37.327268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a9a1d1-71c9-46ce-9e26-4b6013d1560e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:37.341554 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c42e46-897c-4595-93c4-da6b845ab345 admin admin] RBAC: Authorizing `identity:create_grant(project_id=66c178066d9e4b86835ed533fcd680cb, user_id=d90c463f98244312b78e998f4df89188, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:37.345780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c42e46-897c-4595-93c4-da6b845ab345 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:37.348958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 794/1606] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:37 2022] PUT /identity/v3/projects/ba288a6bdc4e41729bd679733eaee61e/users/820f4bb963644499a77fa1f3edd59d08/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:37.351640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a9a1d1-71c9-46ce-9e26-4b6013d1560e admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:37.370365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 813/1607] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:37 2022] PUT /identity/v3/projects/66c178066d9e4b86835ed533fcd680cb/users/d90c463f98244312b78e998f4df89188/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:37.372559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e47af16-b475-40e4-a58c-09bbe6744c6d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.372934 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e47af16-b475-40e4-a58c-09bbe6744c6d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.373311 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e47af16-b475-40e4-a58c-09bbe6744c6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.411278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:37.415487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4e47af16-b475-40e4-a58c-09bbe6744c6d None None] MFA Rules not processed for user `4e6b3e1f654d4365a6b6ead804444986`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:37.418102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4e47af16-b475-40e4-a58c-09bbe6744c6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:37.471467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:37.488376 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a9a1d1-71c9-46ce-9e26-4b6013d1560e admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:37.517046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 814/1608] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:37 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:37.518627 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8859bf6c-6d96-4883-b062-0326fb979afc None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.519286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8859bf6c-6d96-4883-b062-0326fb979afc None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.519730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8859bf6c-6d96-4883-b062-0326fb979afc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.560374 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-8859bf6c-6d96-4883-b062-0326fb979afc None None] MFA Rules not processed for user `fe401c67b1d54c2c86dd1e6f289f5b11`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:37.564450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8859bf6c-6d96-4883-b062-0326fb979afc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:37.610668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4922493-ccf7-45cf-a650-d800be4472e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:37.611484 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4922493-ccf7-45cf-a650-d800be4472e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.611766 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4922493-ccf7-45cf-a650-d800be4472e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.612046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4922493-ccf7-45cf-a650-d800be4472e7 admin admin] PATH_INFO: `/v3/projects/6090ef3dd4494004b0156e49bbeb9273/users/3b0e4c6131674b3799a374aacdec708f/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.632857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4922493-ccf7-45cf-a650-d800be4472e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6090ef3dd4494004b0156e49bbeb9273, user_id=3b0e4c6131674b3799a374aacdec708f, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:37.635724 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4922493-ccf7-45cf-a650-d800be4472e7 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:37.653525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 795/1609] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:37 2022] PUT /identity/v3/projects/6090ef3dd4494004b0156e49bbeb9273/users/3b0e4c6131674b3799a374aacdec708f/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:37.653525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4922493-ccf7-45cf-a650-d800be4472e7 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:37.669930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 815/1610] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:37 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:37.674265 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e043b11-c073-4a93-98f4-abbb4cf12b9b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.674265 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e043b11-c073-4a93-98f4-abbb4cf12b9b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.674265 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e043b11-c073-4a93-98f4-abbb4cf12b9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.700465 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-3e043b11-c073-4a93-98f4-abbb4cf12b9b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:37.702382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-3e043b11-c073-4a93-98f4-abbb4cf12b9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:37.753832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4922493-ccf7-45cf-a650-d800be4472e7 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:37.761580 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c92c15-b35b-4d94-9cee-a35430adb86f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:37.762305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c92c15-b35b-4d94-9cee-a35430adb86f service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.762673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c92c15-b35b-4d94-9cee-a35430adb86f service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.763097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c92c15-b35b-4d94-9cee-a35430adb86f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.784049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 816/1611] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:37.784049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5305ca18-a989-41c8-9ee8-597e4d01da4b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.784049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5305ca18-a989-41c8-9ee8-597e4d01da4b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.784479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5305ca18-a989-41c8-9ee8-597e4d01da4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.829527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-5305ca18-a989-41c8-9ee8-597e4d01da4b None None] MFA Rules not processed for user `d90c463f98244312b78e998f4df89188`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:37.831226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5305ca18-a989-41c8-9ee8-597e4d01da4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:37.872710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c92c15-b35b-4d94-9cee-a35430adb86f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:37.874969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c92c15-b35b-4d94-9cee-a35430adb86f service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:37.883887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:37.892036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 796/1612] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:37.893615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c92c15-b35b-4d94-9cee-a35430adb86f service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:37.909465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c92c15-b35b-4d94-9cee-a35430adb86f service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:37.933412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07981301-5184-46b8-8c23-9e42e1129fac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:37.935324 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07981301-5184-46b8-8c23-9e42e1129fac admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.935733 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07981301-5184-46b8-8c23-9e42e1129fac admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.936380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07981301-5184-46b8-8c23-9e42e1129fac admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:37.937248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07981301-5184-46b8-8c23-9e42e1129fac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:37.939549 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07981301-5184-46b8-8c23-9e42e1129fac admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:37.990879 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 817/1613] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:37 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:37.992611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0929309f-de64-4c8b-b0f0-3632724eb48a None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:37.993159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0929309f-de64-4c8b-b0f0-3632724eb48a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:37.993539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0929309f-de64-4c8b-b0f0-3632724eb48a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:38.008027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 797/1614] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:37 2022] POST /identity/v3/projects => generated 405 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:38.009546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07981301-5184-46b8-8c23-9e42e1129fac admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:38.040105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:38.044505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-0929309f-de64-4c8b-b0f0-3632724eb48a None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:38.048395 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0929309f-de64-4c8b-b0f0-3632724eb48a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:38.162438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07981301-5184-46b8-8c23-9e42e1129fac admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:38.175101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2164e0e0-33e5-4f83-beb0-0fbff20ee1e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:38.177358 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2164e0e0-33e5-4f83-beb0-0fbff20ee1e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:38.177827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2164e0e0-33e5-4f83-beb0-0fbff20ee1e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:38.178413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2164e0e0-33e5-4f83-beb0-0fbff20ee1e8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:38.179805 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2164e0e0-33e5-4f83-beb0-0fbff20ee1e8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:38.182477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2164e0e0-33e5-4f83-beb0-0fbff20ee1e8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:38.197847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 818/1615] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:38.197847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0929309f-de64-4c8b-b0f0-3632724eb48a None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:38.256015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 798/1616] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:38 2022] POST /identity/v3/projects => generated 365 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:38.256015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2164e0e0-33e5-4f83-beb0-0fbff20ee1e8 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:38.325048 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0929309f-de64-4c8b-b0f0-3632724eb48a None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:38.397705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2164e0e0-33e5-4f83-beb0-0fbff20ee1e8 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:38.409994 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d67816-9e7a-4327-8a8c-28986255bb90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:38.410333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d67816-9e7a-4327-8a8c-28986255bb90 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:38.411113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d67816-9e7a-4327-8a8c-28986255bb90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:38.411532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d67816-9e7a-4327-8a8c-28986255bb90 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:38.413477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d67816-9e7a-4327-8a8c-28986255bb90 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:38.418615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d67816-9e7a-4327-8a8c-28986255bb90 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:38.431987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44de5068-b6e5-4d13-8b39-f1b0151b9aa6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:38.432933 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44de5068-b6e5-4d13-8b39-f1b0151b9aa6 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:38.433354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44de5068-b6e5-4d13-8b39-f1b0151b9aa6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:38.433737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44de5068-b6e5-4d13-8b39-f1b0151b9aa6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:38.454654 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44de5068-b6e5-4d13-8b39-f1b0151b9aa6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:38.456436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44de5068-b6e5-4d13-8b39-f1b0151b9aa6 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:38.473677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 799/1617] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:38 2022] POST /identity/v3/projects => generated 357 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:38.474727 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d67816-9e7a-4327-8a8c-28986255bb90 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:38.479707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 819/1618] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:38.480946 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44de5068-b6e5-4d13-8b39-f1b0151b9aa6 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:38.502706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44de5068-b6e5-4d13-8b39-f1b0151b9aa6 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:38.556843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:38.631909 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d67816-9e7a-4327-8a8c-28986255bb90 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:38.636715 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1c55da-ab79-4024-903c-9770eabc2cfc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:38.637649 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1c55da-ab79-4024-903c-9770eabc2cfc service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:38.638038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1c55da-ab79-4024-903c-9770eabc2cfc service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:38.638563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1c55da-ab79-4024-903c-9770eabc2cfc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:38.652815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1c55da-ab79-4024-903c-9770eabc2cfc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:38.654387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1c55da-ab79-4024-903c-9770eabc2cfc service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:38.655309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86833156-3d26-4037-b246-e465bd38feac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:38.656736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86833156-3d26-4037-b246-e465bd38feac admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:38.656736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86833156-3d26-4037-b246-e465bd38feac admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:38.656991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86833156-3d26-4037-b246-e465bd38feac admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:38.657878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86833156-3d26-4037-b246-e465bd38feac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:38.660355 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86833156-3d26-4037-b246-e465bd38feac admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:38.668716 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 820/1619] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:38.669774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1c55da-ab79-4024-903c-9770eabc2cfc service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:38.681119 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1c55da-ab79-4024-903c-9770eabc2cfc service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:38.689780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac67051a-92bc-4282-8ced-a105f10d7e49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:38.690913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac67051a-92bc-4282-8ced-a105f10d7e49 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:38.691806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac67051a-92bc-4282-8ced-a105f10d7e49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:38.692396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac67051a-92bc-4282-8ced-a105f10d7e49 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:38.693433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac67051a-92bc-4282-8ced-a105f10d7e49 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:38.696200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac67051a-92bc-4282-8ced-a105f10d7e49 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:38.700718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 800/1620] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:38 2022] POST /identity/v3/users => generated 309 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:38.701561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86833156-3d26-4037-b246-e465bd38feac admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:38.719425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86833156-3d26-4037-b246-e465bd38feac admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:38.734133 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5477917-cf92-4b89-b7e0-b6b40b1842ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:38.734855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 821/1621] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:38 2022] POST /identity/v3/users => generated 289 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:38.735745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac67051a-92bc-4282-8ced-a105f10d7e49 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:38.738031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5477917-cf92-4b89-b7e0-b6b40b1842ef admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:38.738421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5477917-cf92-4b89-b7e0-b6b40b1842ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:38.738646 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5477917-cf92-4b89-b7e0-b6b40b1842ef admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:38.739317 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5477917-cf92-4b89-b7e0-b6b40b1842ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:38.741425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5477917-cf92-4b89-b7e0-b6b40b1842ef admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:38.752439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac67051a-92bc-4282-8ced-a105f10d7e49 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:38.761523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405201d1-08f4-40da-8397-3b78836c6d39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:38.762186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405201d1-08f4-40da-8397-3b78836c6d39 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:38.762944 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405201d1-08f4-40da-8397-3b78836c6d39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:38.763148 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405201d1-08f4-40da-8397-3b78836c6d39 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:38.766611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:38.771573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405201d1-08f4-40da-8397-3b78836c6d39 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:38.776040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405201d1-08f4-40da-8397-3b78836c6d39 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:38.789906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 801/1622] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:38 2022] POST /identity/v3/users => generated 285 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:38.790738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5477917-cf92-4b89-b7e0-b6b40b1842ef admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:38.796593 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 822/1623] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:38 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:38.797706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405201d1-08f4-40da-8397-3b78836c6d39 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:38.811125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5477917-cf92-4b89-b7e0-b6b40b1842ef admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:38.817061 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405201d1-08f4-40da-8397-3b78836c6d39 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:38.822957 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fd7a7b-21aa-4b8b-b75d-003194f9bc9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:38.824180 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fd7a7b-21aa-4b8b-b75d-003194f9bc9d admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:38.824456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fd7a7b-21aa-4b8b-b75d-003194f9bc9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:38.824673 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fd7a7b-21aa-4b8b-b75d-003194f9bc9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:38.825720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fd7a7b-21aa-4b8b-b75d-003194f9bc9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:38.827587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fd7a7b-21aa-4b8b-b75d-003194f9bc9d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:38.833161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69207ca-afb7-402a-b83b-ea943c2b28b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:38.833788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69207ca-afb7-402a-b83b-ea943c2b28b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:38.834112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69207ca-afb7-402a-b83b-ea943c2b28b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:38.834423 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69207ca-afb7-402a-b83b-ea943c2b28b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:38.835109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69207ca-afb7-402a-b83b-ea943c2b28b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:38.836946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69207ca-afb7-402a-b83b-ea943c2b28b2 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:38.853184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 802/1624] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:38 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:38.854323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69207ca-afb7-402a-b83b-ea943c2b28b2 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:38.858653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 823/1625] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:38 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:38.871954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69207ca-afb7-402a-b83b-ea943c2b28b2 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:38.872494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fd7a7b-21aa-4b8b-b75d-003194f9bc9d admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:38.878277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e42979-133a-46c4-a66a-2f443c192be7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:38.879130 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e42979-133a-46c4-a66a-2f443c192be7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:38.879525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e42979-133a-46c4-a66a-2f443c192be7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:38.879899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e42979-133a-46c4-a66a-2f443c192be7 admin admin] PATH_INFO: `/v3/projects/24df4f3d4adb457684df6aaf1fbfce0a/users/50c11abe889a41e4b406d8ec6427f908/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:38.901361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fd7a7b-21aa-4b8b-b75d-003194f9bc9d admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:38.907281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e42979-133a-46c4-a66a-2f443c192be7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24df4f3d4adb457684df6aaf1fbfce0a, user_id=50c11abe889a41e4b406d8ec6427f908, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:38.910049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3016c53-8042-4c0d-b01e-f218c0c4627c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:38.910790 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3016c53-8042-4c0d-b01e-f218c0c4627c admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:38.911127 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3016c53-8042-4c0d-b01e-f218c0c4627c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:38.911594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3016c53-8042-4c0d-b01e-f218c0c4627c admin admin] PATH_INFO: `/v3/projects/ffb48e2021154333b92ef18b13e7e10f/users/838493de3ac24f63b87d3e216cdffb17/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:38.916223 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e42979-133a-46c4-a66a-2f443c192be7 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:38.939898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3016c53-8042-4c0d-b01e-f218c0c4627c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ffb48e2021154333b92ef18b13e7e10f, user_id=838493de3ac24f63b87d3e216cdffb17, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:38.945658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3016c53-8042-4c0d-b01e-f218c0c4627c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:38.947668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 803/1626] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:38 2022] PUT /identity/v3/projects/24df4f3d4adb457684df6aaf1fbfce0a/users/50c11abe889a41e4b406d8ec6427f908/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:38.948830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e42979-133a-46c4-a66a-2f443c192be7 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:38.987398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 824/1627] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:38 2022] PUT /identity/v3/projects/ffb48e2021154333b92ef18b13e7e10f/users/838493de3ac24f63b87d3e216cdffb17/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:38.989909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3016c53-8042-4c0d-b01e-f218c0c4627c admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:39.095935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e42979-133a-46c4-a66a-2f443c192be7 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:39.167043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3016c53-8042-4c0d-b01e-f218c0c4627c admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:39.182433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978204d6-d371-4efb-8f98-628b58725144 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:39.183139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978204d6-d371-4efb-8f98-628b58725144 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:39.183490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978204d6-d371-4efb-8f98-628b58725144 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:39.183775 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978204d6-d371-4efb-8f98-628b58725144 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:39.184550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978204d6-d371-4efb-8f98-628b58725144 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:39.193187 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978204d6-d371-4efb-8f98-628b58725144 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:39.229464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 825/1628] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:38 2022] GET /identity/v3/roles => generated 1420 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:39.230487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978204d6-d371-4efb-8f98-628b58725144 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:39.245097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b7126d-5985-4b42-9d7f-6a6d926d52c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:39.246627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b7126d-5985-4b42-9d7f-6a6d926d52c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:39.248663 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b7126d-5985-4b42-9d7f-6a6d926d52c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:39.249863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b7126d-5985-4b42-9d7f-6a6d926d52c7 admin admin] PATH_INFO: `/v3/projects/8d330d3d8c024696a812f028d3509bf9/users/05a44c3e1dce43c48b91a53b6f3e15d2/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:39.256535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978204d6-d371-4efb-8f98-628b58725144 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:39.263855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a947ab-0cd7-4092-99fe-19ae945ab525 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:39.264656 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a947ab-0cd7-4092-99fe-19ae945ab525 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:39.265044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a947ab-0cd7-4092-99fe-19ae945ab525 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:39.265960 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a947ab-0cd7-4092-99fe-19ae945ab525 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:39.265960 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a947ab-0cd7-4092-99fe-19ae945ab525 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:39.267752 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a947ab-0cd7-4092-99fe-19ae945ab525 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:39.295695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b7126d-5985-4b42-9d7f-6a6d926d52c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d330d3d8c024696a812f028d3509bf9, user_id=05a44c3e1dce43c48b91a53b6f3e15d2, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:39.302129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b7126d-5985-4b42-9d7f-6a6d926d52c7 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:39.303340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 826/1629] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:39 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:39.304850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a947ab-0cd7-4092-99fe-19ae945ab525 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:39.329355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a947ab-0cd7-4092-99fe-19ae945ab525 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:39.336270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 804/1630] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:38 2022] PUT /identity/v3/projects/8d330d3d8c024696a812f028d3509bf9/users/05a44c3e1dce43c48b91a53b6f3e15d2/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:39.337132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b7126d-5985-4b42-9d7f-6a6d926d52c7 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:39.464862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:39.482688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:39.517283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b7126d-5985-4b42-9d7f-6a6d926d52c7 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:39.520027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2c2e22-591a-42df-abd7-ad692b28e1e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:39.520746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2c2e22-591a-42df-abd7-ad692b28e1e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:39.521022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2c2e22-591a-42df-abd7-ad692b28e1e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:39.521390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2c2e22-591a-42df-abd7-ad692b28e1e8 admin admin] PATH_INFO: `/v3/projects/24df4f3d4adb457684df6aaf1fbfce0a/users/50c11abe889a41e4b406d8ec6427f908/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:39.522911 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e1e262-1ffa-400a-9d25-32affcc78bc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:39.523422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2c2e22-591a-42df-abd7-ad692b28e1e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24df4f3d4adb457684df6aaf1fbfce0a, user_id=50c11abe889a41e4b406d8ec6427f908, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:39.523866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e1e262-1ffa-400a-9d25-32affcc78bc8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:39.524253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e1e262-1ffa-400a-9d25-32affcc78bc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:39.524731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e1e262-1ffa-400a-9d25-32affcc78bc8 admin admin] PATH_INFO: `/v3/projects/ffb48e2021154333b92ef18b13e7e10f/users/838493de3ac24f63b87d3e216cdffb17/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:39.526085 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2c2e22-591a-42df-abd7-ad692b28e1e8 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:39.527059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e1e262-1ffa-400a-9d25-32affcc78bc8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ffb48e2021154333b92ef18b13e7e10f, user_id=838493de3ac24f63b87d3e216cdffb17, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:39.530966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e1e262-1ffa-400a-9d25-32affcc78bc8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:39.550928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 805/1631] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:39 2022] PUT /identity/v3/projects/ffb48e2021154333b92ef18b13e7e10f/users/838493de3ac24f63b87d3e216cdffb17/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:39.552729 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e1e262-1ffa-400a-9d25-32affcc78bc8 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:39.555560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 827/1632] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:39 2022] PUT /identity/v3/projects/24df4f3d4adb457684df6aaf1fbfce0a/users/50c11abe889a41e4b406d8ec6427f908/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:39.563713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503923ed-78d0-4507-92f4-fd4164a5112e None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:39.564010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503923ed-78d0-4507-92f4-fd4164a5112e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:39.564553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503923ed-78d0-4507-92f4-fd4164a5112e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:39.618434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-503923ed-78d0-4507-92f4-fd4164a5112e None None] MFA Rules not processed for user `838493de3ac24f63b87d3e216cdffb17`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:39.619983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-503923ed-78d0-4507-92f4-fd4164a5112e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:39.769870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e1e262-1ffa-400a-9d25-32affcc78bc8 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:39.830913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 828/1633] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:39 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:39.834070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6598a1be-8921-4c36-95c9-e66c09cf2dab None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:39.834531 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6598a1be-8921-4c36-95c9-e66c09cf2dab None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:39.835096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6598a1be-8921-4c36-95c9-e66c09cf2dab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:39.881546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-6598a1be-8921-4c36-95c9-e66c09cf2dab None None] MFA Rules not processed for user `50c11abe889a41e4b406d8ec6427f908`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:39.883787 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6598a1be-8921-4c36-95c9-e66c09cf2dab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:39.925248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6825e9-df6d-46b3-8d83-21e937d215c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:39.926024 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6825e9-df6d-46b3-8d83-21e937d215c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:39.926407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6825e9-df6d-46b3-8d83-21e937d215c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:39.926782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6825e9-df6d-46b3-8d83-21e937d215c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:39.927539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6825e9-df6d-46b3-8d83-21e937d215c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:39.929682 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6825e9-df6d-46b3-8d83-21e937d215c6 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:39.942284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 806/1634] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:39 2022] GET /identity/v3/roles => generated 1420 bytes in 391 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:39.943898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6825e9-df6d-46b3-8d83-21e937d215c6 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:40.077822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6825e9-df6d-46b3-8d83-21e937d215c6 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:40.088136 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 829/1635] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:39 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:40.089281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6598a1be-8921-4c36-95c9-e66c09cf2dab None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:40.093013 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510d7021-5b85-4fd6-bd04-49a195a3735d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:40.094808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510d7021-5b85-4fd6-bd04-49a195a3735d service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:40.095822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510d7021-5b85-4fd6-bd04-49a195a3735d service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:40.096039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510d7021-5b85-4fd6-bd04-49a195a3735d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:40.119876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6598a1be-8921-4c36-95c9-e66c09cf2dab None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:40.121072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510d7021-5b85-4fd6-bd04-49a195a3735d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:40.124577 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510d7021-5b85-4fd6-bd04-49a195a3735d service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:40.132545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01d31b0-eb38-4f8c-b4b9-cacac7cfd2cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:40.138705 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01d31b0-eb38-4f8c-b4b9-cacac7cfd2cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:40.139076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01d31b0-eb38-4f8c-b4b9-cacac7cfd2cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:40.139365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01d31b0-eb38-4f8c-b4b9-cacac7cfd2cf admin admin] PATH_INFO: `/v3/projects/8d330d3d8c024696a812f028d3509bf9/users/05a44c3e1dce43c48b91a53b6f3e15d2/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:40.163204 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01d31b0-eb38-4f8c-b4b9-cacac7cfd2cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d330d3d8c024696a812f028d3509bf9, user_id=05a44c3e1dce43c48b91a53b6f3e15d2, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:40.173964 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01d31b0-eb38-4f8c-b4b9-cacac7cfd2cf admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:40.180742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 807/1636] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:39 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:40.182647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510d7021-5b85-4fd6-bd04-49a195a3735d service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:40.202908 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510d7021-5b85-4fd6-bd04-49a195a3735d service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:40.204295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 830/1637] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:40 2022] PUT /identity/v3/projects/8d330d3d8c024696a812f028d3509bf9/users/05a44c3e1dce43c48b91a53b6f3e15d2/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:40.217484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4292c0f4-ae5e-42a1-8167-69704431ca4e None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:40.218662 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4292c0f4-ae5e-42a1-8167-69704431ca4e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:40.218946 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4292c0f4-ae5e-42a1-8167-69704431ca4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:40.316785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:40.321935 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4292c0f4-ae5e-42a1-8167-69704431ca4e None None] MFA Rules not processed for user `05a44c3e1dce43c48b91a53b6f3e15d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:40.323282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4292c0f4-ae5e-42a1-8167-69704431ca4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:40.382816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d27cfa3-ea91-489a-bbf3-cfc6d22c5266 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:40.383465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27cfa3-ea91-489a-bbf3-cfc6d22c5266 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:40.383740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27cfa3-ea91-489a-bbf3-cfc6d22c5266 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:40.384014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27cfa3-ea91-489a-bbf3-cfc6d22c5266 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:40.405155 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:40.516579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 831/1638] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:40 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 301 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:40.537093 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d27cfa3-ea91-489a-bbf3-cfc6d22c5266 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:40.538725 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d27cfa3-ea91-489a-bbf3-cfc6d22c5266 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:40.553905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 808/1639] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:40 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:40.586655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4292c0f4-ae5e-42a1-8167-69704431ca4e None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:40.604842 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4292c0f4-ae5e-42a1-8167-69704431ca4e None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:40.627482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978d5b93-2748-4a98-9efa-a117f3fd916b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:40.628159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978d5b93-2748-4a98-9efa-a117f3fd916b service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:40.628549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978d5b93-2748-4a98-9efa-a117f3fd916b service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:40.629022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978d5b93-2748-4a98-9efa-a117f3fd916b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:40.656614 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978d5b93-2748-4a98-9efa-a117f3fd916b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:40.658473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978d5b93-2748-4a98-9efa-a117f3fd916b service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:40.672709 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 832/1640] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:40 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:40.937000 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d27cfa3-ea91-489a-bbf3-cfc6d22c5266 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:40.953456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d27cfa3-ea91-489a-bbf3-cfc6d22c5266 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:40.986330 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db65b3f-7b27-4c36-ae7c-c08627026619 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:40.989159 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db65b3f-7b27-4c36-ae7c-c08627026619 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:40.996280 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db65b3f-7b27-4c36-ae7c-c08627026619 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:40.996903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db65b3f-7b27-4c36-ae7c-c08627026619 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:41.150264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db65b3f-7b27-4c36-ae7c-c08627026619 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:41.152272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db65b3f-7b27-4c36-ae7c-c08627026619 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:41.167051 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978d5b93-2748-4a98-9efa-a117f3fd916b service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:41.175473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 809/1641] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:40 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 241 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:41.319644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978d5b93-2748-4a98-9efa-a117f3fd916b service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:41.325495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4279f63-9931-42ab-84ba-ccaa085a99a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:41.326276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4279f63-9931-42ab-84ba-ccaa085a99a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:41.326559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4279f63-9931-42ab-84ba-ccaa085a99a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:41.326836 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4279f63-9931-42ab-84ba-ccaa085a99a0 admin admin] PATH_INFO: `/v3/users/4e6b3e1f654d4365a6b6ead804444986` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:41.330280 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4279f63-9931-42ab-84ba-ccaa085a99a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e6b3e1f654d4365a6b6ead804444986)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:41.334209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4279f63-9931-42ab-84ba-ccaa085a99a0 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:41.417703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db65b3f-7b27-4c36-ae7c-c08627026619 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:41.447187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db65b3f-7b27-4c36-ae7c-c08627026619 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:41.450108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c4279f63-9931-42ab-84ba-ccaa085a99a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e6b3e1f654d4365a6b6ead804444986'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:41.452863 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c4279f63-9931-42ab-84ba-ccaa085a99a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e6b3e1f654d4365a6b6ead804444986'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:41.458159 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:41.459026 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:41.459459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:41.459977 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] PATH_INFO: `/v3/users/fe401c67b1d54c2c86dd1e6f289f5b11` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:41.465723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe401c67b1d54c2c86dd1e6f289f5b11)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:41.465921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c4279f63-9931-42ab-84ba-ccaa085a99a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e6b3e1f654d4365a6b6ead804444986'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:41.466986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c4279f63-9931-42ab-84ba-ccaa085a99a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e6b3e1f654d4365a6b6ead804444986'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:41.467254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: WARNING py.warnings [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:14:41.467254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: warnings.warn(msg) Sep 28 12:14:41.467254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: Sep 28 12:14:41.487088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c4279f63-9931-42ab-84ba-ccaa085a99a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e6b3e1f654d4365a6b6ead804444986'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:41.487498 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:41.558183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 833/1642] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:14:41 2022] DELETE /identity/v3/users/4e6b3e1f654d4365a6b6ead804444986 => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:41.558183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c595578e-0886-4e27-9019-1ee072fbfd57 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:41.558183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c595578e-0886-4e27-9019-1ee072fbfd57 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:41.558183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c595578e-0886-4e27-9019-1ee072fbfd57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:41.586842 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-c595578e-0886-4e27-9019-1ee072fbfd57 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:41.588167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c595578e-0886-4e27-9019-1ee072fbfd57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:41.656713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:41.737762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe401c67b1d54c2c86dd1e6f289f5b11'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:41.738930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe401c67b1d54c2c86dd1e6f289f5b11'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:41.739571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe401c67b1d54c2c86dd1e6f289f5b11'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:41.753315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe401c67b1d54c2c86dd1e6f289f5b11'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:41.781362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 834/1643] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:14:41.785105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:41.788557 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe401c67b1d54c2c86dd1e6f289f5b11'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:41.814986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 810/1644] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:14:41 2022] DELETE /identity/v3/users/fe401c67b1d54c2c86dd1e6f289f5b11 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:41.819398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc338401-37a8-40a3-b077-179902b89c2f None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:41.819799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc338401-37a8-40a3-b077-179902b89c2f None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:41.820211 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc338401-37a8-40a3-b077-179902b89c2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:41.850956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-bc338401-37a8-40a3-b077-179902b89c2f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:41.853502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-bc338401-37a8-40a3-b077-179902b89c2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:41.956666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 835/1645] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:42.029778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:42.032284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:42.088603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc338401-37a8-40a3-b077-179902b89c2f None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:42.090326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-bc338401-37a8-40a3-b077-179902b89c2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:42.158846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fe1064-bab7-40e9-9018-0ea7040d4d12 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:42.163878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41445301-998b-4b0e-ba56-8132791d4aa0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:42.172398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41445301-998b-4b0e-ba56-8132791d4aa0 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:42.172733 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41445301-998b-4b0e-ba56-8132791d4aa0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:42.173028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41445301-998b-4b0e-ba56-8132791d4aa0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:42.200423 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41445301-998b-4b0e-ba56-8132791d4aa0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:42.202316 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41445301-998b-4b0e-ba56-8132791d4aa0 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:42.215683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 811/1646] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:42.235057 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc338401-37a8-40a3-b077-179902b89c2f None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:42.239417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25152f7-f2aa-4b6e-a634-045f55282046 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:42.240212 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25152f7-f2aa-4b6e-a634-045f55282046 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:42.240685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25152f7-f2aa-4b6e-a634-045f55282046 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:42.241283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25152f7-f2aa-4b6e-a634-045f55282046 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:42.260273 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25152f7-f2aa-4b6e-a634-045f55282046 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:42.261896 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25152f7-f2aa-4b6e-a634-045f55282046 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:42.277910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 836/1647] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:14:43.505991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41445301-998b-4b0e-ba56-8132791d4aa0 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:43.507770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-41445301-998b-4b0e-ba56-8132791d4aa0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:43.526518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41445301-998b-4b0e-ba56-8132791d4aa0 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:43.532091 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:43.532989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:43.533544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:43.533985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] PATH_INFO: `/v3/projects/4755e00b2b02475ea04b589bb7865d92` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:43.535320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] RBAC: Authorizing `identity:delete_project(project_id=4755e00b2b02475ea04b589bb7865d92)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:43.537691 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:43.591949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4755e00b2b02475ea04b589bb7865d92', 'request_id': 'req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:43.600071 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4755e00b2b02475ea04b589bb7865d92', 'request_id': 'req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:43.611420 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4755e00b2b02475ea04b589bb7865d92', 'request_id': 'req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:43.633967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] The token cache is being invalidate because project 4755e00b2b02475ea04b589bb7865d92 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:14:43.634641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:43.635614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:43.638416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 812/1648] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:14:43 2022] DELETE /identity/v3/projects/4755e00b2b02475ea04b589bb7865d92 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:43.643479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25152f7-f2aa-4b6e-a634-045f55282046 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:43.667030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25152f7-f2aa-4b6e-a634-045f55282046 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:43.785699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44147980-8b08-4c48-bb6d-624d3487e611 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:43.786354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44147980-8b08-4c48-bb6d-624d3487e611 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:43.786634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44147980-8b08-4c48-bb6d-624d3487e611 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:43.786909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44147980-8b08-4c48-bb6d-624d3487e611 admin admin] PATH_INFO: `/v3/users/50c11abe889a41e4b406d8ec6427f908` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:43.788071 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44147980-8b08-4c48-bb6d-624d3487e611 admin admin] RBAC: Authorizing `identity:delete_user(user_id=50c11abe889a41e4b406d8ec6427f908)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:43.790039 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44147980-8b08-4c48-bb6d-624d3487e611 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:43.823119 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:43.898141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-44147980-8b08-4c48-bb6d-624d3487e611 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '50c11abe889a41e4b406d8ec6427f908'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:43.898851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-44147980-8b08-4c48-bb6d-624d3487e611 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '50c11abe889a41e4b406d8ec6427f908'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:43.915358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-44147980-8b08-4c48-bb6d-624d3487e611 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '50c11abe889a41e4b406d8ec6427f908'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:43.916153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-44147980-8b08-4c48-bb6d-624d3487e611 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '50c11abe889a41e4b406d8ec6427f908'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:43.943926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-44147980-8b08-4c48-bb6d-624d3487e611 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '50c11abe889a41e4b406d8ec6427f908'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:43.964988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 837/1649] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:14:43 2022] DELETE /identity/v3/users/50c11abe889a41e4b406d8ec6427f908 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:44.456341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:44.458013 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:44.484516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c45bce-232c-4e77-a9cf-0ac9e01d222d admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:44.636212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c017172-2335-46e9-b482-f169f6081e0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:44.637433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c017172-2335-46e9-b482-f169f6081e0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:44.638851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c017172-2335-46e9-b482-f169f6081e0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:44.639706 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c017172-2335-46e9-b482-f169f6081e0b admin admin] PATH_INFO: `/v3/projects/24df4f3d4adb457684df6aaf1fbfce0a` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:44.645683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c017172-2335-46e9-b482-f169f6081e0b admin admin] RBAC: Authorizing `identity:delete_project(project_id=24df4f3d4adb457684df6aaf1fbfce0a)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:44.651865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c017172-2335-46e9-b482-f169f6081e0b admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:44.724650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:44.824760 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-4c017172-2335-46e9-b482-f169f6081e0b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '24df4f3d4adb457684df6aaf1fbfce0a', 'request_id': 'req-4c017172-2335-46e9-b482-f169f6081e0b'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:44.843005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44147980-8b08-4c48-bb6d-624d3487e611 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:44.845922 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-44147980-8b08-4c48-bb6d-624d3487e611 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:44.867410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-4c017172-2335-46e9-b482-f169f6081e0b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '24df4f3d4adb457684df6aaf1fbfce0a', 'request_id': 'req-4c017172-2335-46e9-b482-f169f6081e0b'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:44.886980 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44147980-8b08-4c48-bb6d-624d3487e611 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:44.889307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-4c017172-2335-46e9-b482-f169f6081e0b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '24df4f3d4adb457684df6aaf1fbfce0a', 'request_id': 'req-4c017172-2335-46e9-b482-f169f6081e0b'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:44.931390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-4c017172-2335-46e9-b482-f169f6081e0b admin admin] The token cache is being invalidate because project 24df4f3d4adb457684df6aaf1fbfce0a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:14:44.931929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-4c017172-2335-46e9-b482-f169f6081e0b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:44.934305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-4c017172-2335-46e9-b482-f169f6081e0b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:44.938354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 813/1650] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:14:44 2022] DELETE /identity/v3/projects/24df4f3d4adb457684df6aaf1fbfce0a => generated 0 bytes in 482 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:44.943063 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b45fd-5b10-4b42-9471-a7bea644d08e None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:44.943353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b45fd-5b10-4b42-9471-a7bea644d08e None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:44.943652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b45fd-5b10-4b42-9471-a7bea644d08e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:45.000857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-457b45fd-5b10-4b42-9471-a7bea644d08e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:45.002138 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-457b45fd-5b10-4b42-9471-a7bea644d08e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:45.042030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:45.042806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:45.042953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:45.043398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] PATH_INFO: `/v3/projects/3992ec06d59148758b6883f90f0b4f06` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:45.044971 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] RBAC: Authorizing `identity:delete_project(project_id=3992ec06d59148758b6883f90f0b4f06)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:45.046232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: WARNING py.warnings [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 28 12:14:45.046232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: warnings.warn(msg) Sep 28 12:14:45.046232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: Sep 28 12:14:45.047764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:45.211711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 814/1651] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:14:45.221372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1418cb2-22ad-40f3-8f4e-b957ea4584e5 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:45.222048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1418cb2-22ad-40f3-8f4e-b957ea4584e5 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:45.222457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1418cb2-22ad-40f3-8f4e-b957ea4584e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:45.224338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3992ec06d59148758b6883f90f0b4f06', 'request_id': 'req-a6900989-7ca6-477c-9adf-c72c81a3ec7a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:45.241644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3992ec06d59148758b6883f90f0b4f06', 'request_id': 'req-a6900989-7ca6-477c-9adf-c72c81a3ec7a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:45.271940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3992ec06d59148758b6883f90f0b4f06', 'request_id': 'req-a6900989-7ca6-477c-9adf-c72c81a3ec7a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:45.301160 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-b1418cb2-22ad-40f3-8f4e-b957ea4584e5 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:45.307839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b1418cb2-22ad-40f3-8f4e-b957ea4584e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:45.352918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] The token cache is being invalidate because project 3992ec06d59148758b6883f90f0b4f06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:14:45.354466 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:45.359165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:45.387459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 838/1652] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:14:44 2022] DELETE /identity/v3/projects/3992ec06d59148758b6883f90f0b4f06 => generated 0 bytes in 545 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:45.393646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:45.400251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:45.499202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:45.577720 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 815/1653] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 354 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:14:45.579715 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1418cb2-22ad-40f3-8f4e-b957ea4584e5 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:45.598204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1418cb2-22ad-40f3-8f4e-b957ea4584e5 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:45.602814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6900989-7ca6-477c-9adf-c72c81a3ec7a admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:45.609140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bb108b-c9f3-4e69-8a12-c3ac238edbde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:45.609783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb108b-c9f3-4e69-8a12-c3ac238edbde admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:45.613214 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb108b-c9f3-4e69-8a12-c3ac238edbde admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:45.613542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb108b-c9f3-4e69-8a12-c3ac238edbde admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:45.615784 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bb108b-c9f3-4e69-8a12-c3ac238edbde admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:14:45.616101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bb108b-c9f3-4e69-8a12-c3ac238edbde admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:45.617790 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bb108b-c9f3-4e69-8a12-c3ac238edbde admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:45.626899 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e16706-89f7-46e3-9f33-ace9b6433df8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:45.627575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e16706-89f7-46e3-9f33-ace9b6433df8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:45.628417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e16706-89f7-46e3-9f33-ace9b6433df8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:45.628702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e16706-89f7-46e3-9f33-ace9b6433df8 admin admin] PATH_INFO: `/v3/users/838493de3ac24f63b87d3e216cdffb17` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:45.635974 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e16706-89f7-46e3-9f33-ace9b6433df8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=838493de3ac24f63b87d3e216cdffb17)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:45.645853 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e16706-89f7-46e3-9f33-ace9b6433df8 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:45.671508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 816/1654] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:14:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:14:45.671981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bb108b-c9f3-4e69-8a12-c3ac238edbde admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:45.708757 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bb108b-c9f3-4e69-8a12-c3ac238edbde admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:45.727916 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9564b240-afac-40fe-854e-d0cd8a69ec7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:45.730176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9564b240-afac-40fe-854e-d0cd8a69ec7a admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:45.730595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9564b240-afac-40fe-854e-d0cd8a69ec7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:45.731367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9564b240-afac-40fe-854e-d0cd8a69ec7a admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:45.732282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9564b240-afac-40fe-854e-d0cd8a69ec7a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:45.745541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9564b240-afac-40fe-854e-d0cd8a69ec7a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:45.806720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-57e16706-89f7-46e3-9f33-ace9b6433df8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '838493de3ac24f63b87d3e216cdffb17'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:45.807592 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-57e16706-89f7-46e3-9f33-ace9b6433df8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '838493de3ac24f63b87d3e216cdffb17'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:45.832906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-57e16706-89f7-46e3-9f33-ace9b6433df8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '838493de3ac24f63b87d3e216cdffb17'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:45.835808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-57e16706-89f7-46e3-9f33-ace9b6433df8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '838493de3ac24f63b87d3e216cdffb17'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:45.871727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-57e16706-89f7-46e3-9f33-ace9b6433df8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '838493de3ac24f63b87d3e216cdffb17'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:45.878360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 817/1655] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:45 2022] POST /identity/v3/projects => generated 369 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:45.882061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9564b240-afac-40fe-854e-d0cd8a69ec7a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:45.887292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9564b240-afac-40fe-854e-d0cd8a69ec7a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:45.906921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 839/1656] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:14:45 2022] DELETE /identity/v3/users/838493de3ac24f63b87d3e216cdffb17 => generated 0 bytes in 512 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:45.950641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:46.107184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9564b240-afac-40fe-854e-d0cd8a69ec7a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:46.241380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4a8e71-50d1-493b-a48c-6e0bdd4fef05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:46.244195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4a8e71-50d1-493b-a48c-6e0bdd4fef05 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:46.244730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4a8e71-50d1-493b-a48c-6e0bdd4fef05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:46.245126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4a8e71-50d1-493b-a48c-6e0bdd4fef05 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:46.246085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4a8e71-50d1-493b-a48c-6e0bdd4fef05 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:46.249266 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4a8e71-50d1-493b-a48c-6e0bdd4fef05 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:46.301878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e16706-89f7-46e3-9f33-ace9b6433df8 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:46.311066 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 818/1657] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:45 2022] POST /identity/v3/users => generated 285 bytes in 427 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:14:46.318114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e16706-89f7-46e3-9f33-ace9b6433df8 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:46.331556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f65d860-b721-4374-93d0-bf6a46970594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:46.334308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f65d860-b721-4374-93d0-bf6a46970594 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:46.337007 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f65d860-b721-4374-93d0-bf6a46970594 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:46.337917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f65d860-b721-4374-93d0-bf6a46970594 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:46.338634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f65d860-b721-4374-93d0-bf6a46970594 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:46.345580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f65d860-b721-4374-93d0-bf6a46970594 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:46.363323 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 840/1658] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:46 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:46.367320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4a8e71-50d1-493b-a48c-6e0bdd4fef05 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:46.379860 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4a8e71-50d1-493b-a48c-6e0bdd4fef05 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:46.393247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bff47a2-b34b-469c-beaa-9dd83bb88ffa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:46.393906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bff47a2-b34b-469c-beaa-9dd83bb88ffa admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:46.394743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bff47a2-b34b-469c-beaa-9dd83bb88ffa admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:46.394743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bff47a2-b34b-469c-beaa-9dd83bb88ffa admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:46.395083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bff47a2-b34b-469c-beaa-9dd83bb88ffa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:46.398165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bff47a2-b34b-469c-beaa-9dd83bb88ffa admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:46.423871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 819/1659] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:46 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:46.428161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f65d860-b721-4374-93d0-bf6a46970594 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:46.451623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f65d860-b721-4374-93d0-bf6a46970594 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:46.464367 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76360c3-3872-4ad0-a616-22e7758fb31f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:46.465067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76360c3-3872-4ad0-a616-22e7758fb31f admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:46.465348 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76360c3-3872-4ad0-a616-22e7758fb31f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:46.466224 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76360c3-3872-4ad0-a616-22e7758fb31f admin admin] PATH_INFO: `/v3/projects/8570c618753b487d8a0f9e1a7c5348f4/users/6ecfc15fffa044f68a092a88b9c3dfeb/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:46.517393 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76360c3-3872-4ad0-a616-22e7758fb31f admin admin] RBAC: Authorizing `identity:create_grant(project_id=8570c618753b487d8a0f9e1a7c5348f4, user_id=6ecfc15fffa044f68a092a88b9c3dfeb, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:46.527608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76360c3-3872-4ad0-a616-22e7758fb31f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:46.568851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 841/1660] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:46 2022] PUT /identity/v3/projects/8570c618753b487d8a0f9e1a7c5348f4/users/6ecfc15fffa044f68a092a88b9c3dfeb/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:46.576432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a31f25-a96f-43ce-a802-eaf3a646eb98 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:46.577313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a31f25-a96f-43ce-a802-eaf3a646eb98 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:46.577712 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a31f25-a96f-43ce-a802-eaf3a646eb98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:46.675431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-82a31f25-a96f-43ce-a802-eaf3a646eb98 None None] MFA Rules not processed for user `6ecfc15fffa044f68a092a88b9c3dfeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:46.684061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-82a31f25-a96f-43ce-a802-eaf3a646eb98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:46.830527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76360c3-3872-4ad0-a616-22e7758fb31f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:46.831272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 820/1661] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:46 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:46.981698 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76360c3-3872-4ad0-a616-22e7758fb31f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:46.993111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75435723-bd18-44bc-b308-cb57b478798b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:46.994386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75435723-bd18-44bc-b308-cb57b478798b admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:46.995283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75435723-bd18-44bc-b308-cb57b478798b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:46.996630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75435723-bd18-44bc-b308-cb57b478798b admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:46.998679 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75435723-bd18-44bc-b308-cb57b478798b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:47.003544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75435723-bd18-44bc-b308-cb57b478798b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:47.035560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:47.071455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 842/1662] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:46 2022] POST /identity/v3/projects => generated 369 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:47.074149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a31f25-a96f-43ce-a802-eaf3a646eb98 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:47.103790 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a31f25-a96f-43ce-a802-eaf3a646eb98 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:47.161551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:47.188166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75435723-bd18-44bc-b308-cb57b478798b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:47.190874 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-75435723-bd18-44bc-b308-cb57b478798b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:47.303664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ea4c24-f2a1-429c-a092-7dc8078e4d61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:47.304540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ea4c24-f2a1-429c-a092-7dc8078e4d61 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:47.304831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ea4c24-f2a1-429c-a092-7dc8078e4d61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:47.305161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ea4c24-f2a1-429c-a092-7dc8078e4d61 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:47.306343 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ea4c24-f2a1-429c-a092-7dc8078e4d61 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:47.308820 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ea4c24-f2a1-429c-a092-7dc8078e4d61 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:47.401722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 821/1663] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:47 2022] POST /identity/v3/users => generated 291 bytes in 328 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:47.409404 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ea4c24-f2a1-429c-a092-7dc8078e4d61 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:47.443731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ea4c24-f2a1-429c-a092-7dc8078e4d61 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:47.470888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75435723-bd18-44bc-b308-cb57b478798b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:47.473165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10eaad2e-20bd-40c8-826f-c4ef838d28dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:47.473792 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10eaad2e-20bd-40c8-826f-c4ef838d28dd admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:47.474067 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10eaad2e-20bd-40c8-826f-c4ef838d28dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:47.474338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10eaad2e-20bd-40c8-826f-c4ef838d28dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:47.475002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10eaad2e-20bd-40c8-826f-c4ef838d28dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:47.476653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10eaad2e-20bd-40c8-826f-c4ef838d28dd admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:47.509570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 822/1664] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:47 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:47.510520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10eaad2e-20bd-40c8-826f-c4ef838d28dd admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:47.513011 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:47.513895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:47.514547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:47.525208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97 admin admin] PATH_INFO: `/v3/projects/ffb48e2021154333b92ef18b13e7e10f` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:47.528438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10eaad2e-20bd-40c8-826f-c4ef838d28dd admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:47.537286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ffb48e2021154333b92ef18b13e7e10f)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:47.537560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ad3874-40e6-467c-8862-f7a967c51054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:47.538250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ad3874-40e6-467c-8862-f7a967c51054 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:47.538525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ad3874-40e6-467c-8862-f7a967c51054 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:47.538796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ad3874-40e6-467c-8862-f7a967c51054 admin admin] PATH_INFO: `/v3/projects/c3ff76027be148cf81f57f8d1ccb930f/users/cafd469d08d0431586c3ef4d273f9b46/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:47.543153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:47.566449 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ad3874-40e6-467c-8862-f7a967c51054 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3ff76027be148cf81f57f8d1ccb930f, user_id=cafd469d08d0431586c3ef4d273f9b46, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:47.577916 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ad3874-40e6-467c-8862-f7a967c51054 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:47.617687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 823/1665] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:47 2022] PUT /identity/v3/projects/c3ff76027be148cf81f57f8d1ccb930f/users/cafd469d08d0431586c3ef4d273f9b46/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:47.622116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ad3874-40e6-467c-8862-f7a967c51054 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:47.740604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ffb48e2021154333b92ef18b13e7e10f', 'request_id': 'req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:47.746474 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ad3874-40e6-467c-8862-f7a967c51054 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:47.757015 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ffb48e2021154333b92ef18b13e7e10f', 'request_id': 'req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:47.789105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ffb48e2021154333b92ef18b13e7e10f', 'request_id': 'req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:47.825433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97 admin admin] The token cache is being invalidate because project ffb48e2021154333b92ef18b13e7e10f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:14:47.825433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:47.826050 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d4e493c5-3ea3-48d6-ae49-ceab9ee3ef97 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:47.832187 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 843/1666] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:14:47 2022] DELETE /identity/v3/projects/ffb48e2021154333b92ef18b13e7e10f => generated 0 bytes in 647 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:47.836817 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d72af8-973f-467b-8eb1-644cc513b65f None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:47.837476 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d72af8-973f-467b-8eb1-644cc513b65f None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:47.837476 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d72af8-973f-467b-8eb1-644cc513b65f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:47.913126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-73d72af8-973f-467b-8eb1-644cc513b65f None None] MFA Rules not processed for user `956c7f92a4a747dba49e3f3a527bba74`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:47.914373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-73d72af8-973f-467b-8eb1-644cc513b65f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:47.937227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b923a2fe-d009-4d8c-90cb-1f7c1f1b5fdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:47.939331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b923a2fe-d009-4d8c-90cb-1f7c1f1b5fdf admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:47.941658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b923a2fe-d009-4d8c-90cb-1f7c1f1b5fdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:47.942345 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b923a2fe-d009-4d8c-90cb-1f7c1f1b5fdf admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:47.943425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b923a2fe-d009-4d8c-90cb-1f7c1f1b5fdf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:47.949987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b923a2fe-d009-4d8c-90cb-1f7c1f1b5fdf admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:48.000366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 824/1667] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:47 2022] GET /identity/v3/roles => generated 1420 bytes in 372 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:48.000366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46938ac9-7f21-4f66-8a69-482d71d34f33 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:48.000366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46938ac9-7f21-4f66-8a69-482d71d34f33 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:48.000366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46938ac9-7f21-4f66-8a69-482d71d34f33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:48.040166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-46938ac9-7f21-4f66-8a69-482d71d34f33 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:48.041825 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-46938ac9-7f21-4f66-8a69-482d71d34f33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:48.057378 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 825/1668] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:48.058403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46938ac9-7f21-4f66-8a69-482d71d34f33 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:48.065963 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-46938ac9-7f21-4f66-8a69-482d71d34f33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:48.074821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 844/1669] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:47 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:48.077257 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ede59a-f33d-43d1-a571-c13c18995e88 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:48.077686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ede59a-f33d-43d1-a571-c13c18995e88 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:48.078096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ede59a-f33d-43d1-a571-c13c18995e88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:48.089185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46938ac9-7f21-4f66-8a69-482d71d34f33 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:48.096457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07917861-8071-44d2-82dd-9fe39b6b16bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:48.097271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07917861-8071-44d2-82dd-9fe39b6b16bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:48.097707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07917861-8071-44d2-82dd-9fe39b6b16bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:48.098391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07917861-8071-44d2-82dd-9fe39b6b16bf admin admin] PATH_INFO: `/v3/projects/c3ff76027be148cf81f57f8d1ccb930f/users/cafd469d08d0431586c3ef4d273f9b46/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:48.102096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-90ede59a-f33d-43d1-a571-c13c18995e88 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:48.103309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-90ede59a-f33d-43d1-a571-c13c18995e88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:48.106176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07917861-8071-44d2-82dd-9fe39b6b16bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3ff76027be148cf81f57f8d1ccb930f, user_id=cafd469d08d0431586c3ef4d273f9b46, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:48.108971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07917861-8071-44d2-82dd-9fe39b6b16bf admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:48.121612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 845/1670] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:48.123147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14cde17-389b-47c1-b4fe-21cabe7a3be3 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:48.124646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14cde17-389b-47c1-b4fe-21cabe7a3be3 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:48.126261 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14cde17-389b-47c1-b4fe-21cabe7a3be3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:48.140020 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 826/1671] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:48 2022] PUT /identity/v3/projects/c3ff76027be148cf81f57f8d1ccb930f/users/cafd469d08d0431586c3ef4d273f9b46/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:48.141775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07917861-8071-44d2-82dd-9fe39b6b16bf admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:48.152546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:48.153670 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:48.173804 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-f14cde17-389b-47c1-b4fe-21cabe7a3be3 None None] MFA Rules not processed for user `956c7f92a4a747dba49e3f3a527bba74`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:48.176002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f14cde17-389b-47c1-b4fe-21cabe7a3be3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:48.305854 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 846/1672] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:48 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:48.308824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f34d260-3058-4903-926d-c928deb11f2c None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:48.309184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f34d260-3058-4903-926d-c928deb11f2c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:48.320778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f34d260-3058-4903-926d-c928deb11f2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:48.330026 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07917861-8071-44d2-82dd-9fe39b6b16bf admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:48.348404 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c9d145-b702-4334-8471-c18335abc33f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:48.349375 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c9d145-b702-4334-8471-c18335abc33f admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:48.349375 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c9d145-b702-4334-8471-c18335abc33f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:48.349846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c9d145-b702-4334-8471-c18335abc33f admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:48.361471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c9d145-b702-4334-8471-c18335abc33f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:14:48.362513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c9d145-b702-4334-8471-c18335abc33f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:48.366383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c9d145-b702-4334-8471-c18335abc33f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:48.382741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 827/1673] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:14:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:14:48.387026 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c9d145-b702-4334-8471-c18335abc33f admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:48.402274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c9d145-b702-4334-8471-c18335abc33f admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:48.407541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4f34d260-3058-4903-926d-c928deb11f2c None None] MFA Rules not processed for user `cafd469d08d0431586c3ef4d273f9b46`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:48.412604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4f34d260-3058-4903-926d-c928deb11f2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:48.413801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbe919b-aeb2-4108-922e-bbbbfa742eaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:48.415209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbe919b-aeb2-4108-922e-bbbbfa742eaa admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:48.415645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbe919b-aeb2-4108-922e-bbbbfa742eaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:48.416235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbe919b-aeb2-4108-922e-bbbbfa742eaa admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:48.417098 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbe919b-aeb2-4108-922e-bbbbfa742eaa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:48.420058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbe919b-aeb2-4108-922e-bbbbfa742eaa admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:48.493399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 828/1674] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:48 2022] POST /identity/v3/projects => generated 377 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:48.497791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20660109-31c8-4fe6-aad3-40092f914896 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:48.498087 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20660109-31c8-4fe6-aad3-40092f914896 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:48.498367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20660109-31c8-4fe6-aad3-40092f914896 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:48.503315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 829/1675] 10.222.0.26 () {54 vars in 961 bytes} [Wed Sep 28 12:14:48 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:14:48.504154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20660109-31c8-4fe6-aad3-40092f914896 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:48.583755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 847/1676] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:48 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:48.583755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f7443b-b8d3-42e0-b163-18461e8ee67a None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:48.583755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f7443b-b8d3-42e0-b163-18461e8ee67a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:48.583755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f7443b-b8d3-42e0-b163-18461e8ee67a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:48.662703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20660109-31c8-4fe6-aad3-40092f914896 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:48.675197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-39f7443b-b8d3-42e0-b163-18461e8ee67a None None] MFA Rules not processed for user `8bf969348e7a4158acf8fb789430b0e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:48.676478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-39f7443b-b8d3-42e0-b163-18461e8ee67a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:48.677185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79502ce8-d678-4e16-9ef9-fc038742a04e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:48.677854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79502ce8-d678-4e16-9ef9-fc038742a04e admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:48.678096 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79502ce8-d678-4e16-9ef9-fc038742a04e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:48.678365 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79502ce8-d678-4e16-9ef9-fc038742a04e admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:48.679006 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79502ce8-d678-4e16-9ef9-fc038742a04e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:48.683284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79502ce8-d678-4e16-9ef9-fc038742a04e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:48.743529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 830/1677] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:48 2022] POST /identity/v3/users => generated 289 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:48.747458 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79502ce8-d678-4e16-9ef9-fc038742a04e admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:48.750603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-79502ce8-d678-4e16-9ef9-fc038742a04e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:48.841410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 848/1678] 10.222.0.26 () {58 vars in 1058 bytes} [Wed Sep 28 12:14:48 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:48.845260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f7443b-b8d3-42e0-b163-18461e8ee67a None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:48.865308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f7443b-b8d3-42e0-b163-18461e8ee67a None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:48.876213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb30c97e-8720-4624-88cd-eb56b586454f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:48.876903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb30c97e-8720-4624-88cd-eb56b586454f admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:48.877179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb30c97e-8720-4624-88cd-eb56b586454f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:48.877871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb30c97e-8720-4624-88cd-eb56b586454f admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:48.878269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb30c97e-8720-4624-88cd-eb56b586454f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:48.880341 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb30c97e-8720-4624-88cd-eb56b586454f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:48.918933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79502ce8-d678-4e16-9ef9-fc038742a04e admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:48.926222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 849/1679] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:48 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:48.927506 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb30c97e-8720-4624-88cd-eb56b586454f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:48.930483 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96947e08-3134-40a7-aea8-55a9c5fc4e14 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:48.931415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96947e08-3134-40a7-aea8-55a9c5fc4e14 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:48.935232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96947e08-3134-40a7-aea8-55a9c5fc4e14 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:48.935779 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96947e08-3134-40a7-aea8-55a9c5fc4e14 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:48.954206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:48.963088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb30c97e-8720-4624-88cd-eb56b586454f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:48.970781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56da8768-80c0-48fd-8faa-1a2c2894e6a6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:48.971543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56da8768-80c0-48fd-8faa-1a2c2894e6a6 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:48.971827 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56da8768-80c0-48fd-8faa-1a2c2894e6a6 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:48.972103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56da8768-80c0-48fd-8faa-1a2c2894e6a6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:48.983216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:49.055008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96947e08-3134-40a7-aea8-55a9c5fc4e14 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:49.056801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96947e08-3134-40a7-aea8-55a9c5fc4e14 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:49.078800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 831/1680] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:48 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:49.080473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56da8768-80c0-48fd-8faa-1a2c2894e6a6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:49.083047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56da8768-80c0-48fd-8faa-1a2c2894e6a6 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:49.101182 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 850/1681] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:48 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:49.104963 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96947e08-3134-40a7-aea8-55a9c5fc4e14 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:49.124014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96947e08-3134-40a7-aea8-55a9c5fc4e14 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:49.130647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfaee15-d908-4d68-8967-9c0119b76401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:49.133794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfaee15-d908-4d68-8967-9c0119b76401 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:49.134199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfaee15-d908-4d68-8967-9c0119b76401 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:49.134524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfaee15-d908-4d68-8967-9c0119b76401 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:49.135326 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfaee15-d908-4d68-8967-9c0119b76401 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:49.137587 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfaee15-d908-4d68-8967-9c0119b76401 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:49.154401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 832/1682] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:49 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:49.155768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56da8768-80c0-48fd-8faa-1a2c2894e6a6 service nova] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:49.166653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfaee15-d908-4d68-8967-9c0119b76401 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:49.171978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1dfaee15-d908-4d68-8967-9c0119b76401 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:49.172502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56da8768-80c0-48fd-8faa-1a2c2894e6a6 service nova] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:49.179322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42200f1f-c49d-425f-a9ee-3bd96f69d479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:49.180551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42200f1f-c49d-425f-a9ee-3bd96f69d479 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:49.181044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42200f1f-c49d-425f-a9ee-3bd96f69d479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:49.181521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42200f1f-c49d-425f-a9ee-3bd96f69d479 admin admin] PATH_INFO: `/v3/projects/c9277e00edf8427c942ade41ca99e5bf/users/3b24c0835d8d4d2aa39940b9414a3da1/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:49.188969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfaee15-d908-4d68-8967-9c0119b76401 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:49.193947 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15eeac03-50e9-4238-8f64-e3129f80933e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:49.195134 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eeac03-50e9-4238-8f64-e3129f80933e service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:49.195372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eeac03-50e9-4238-8f64-e3129f80933e service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:49.195708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eeac03-50e9-4238-8f64-e3129f80933e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:49.197809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-15eeac03-50e9-4238-8f64-e3129f80933e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:49.210191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42200f1f-c49d-425f-a9ee-3bd96f69d479 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9277e00edf8427c942ade41ca99e5bf, user_id=3b24c0835d8d4d2aa39940b9414a3da1, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:49.215710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42200f1f-c49d-425f-a9ee-3bd96f69d479 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:49.241120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 851/1683] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:49 2022] PUT /identity/v3/projects/c9277e00edf8427c942ade41ca99e5bf/users/3b24c0835d8d4d2aa39940b9414a3da1/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:49.251938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289315f3-d6bb-46e7-8e88-bd799014821a None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:49.253863 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289315f3-d6bb-46e7-8e88-bd799014821a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:49.254372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289315f3-d6bb-46e7-8e88-bd799014821a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:49.308215 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15eeac03-50e9-4238-8f64-e3129f80933e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:49.310395 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15eeac03-50e9-4238-8f64-e3129f80933e service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:49.323346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-289315f3-d6bb-46e7-8e88-bd799014821a None None] MFA Rules not processed for user `3b24c0835d8d4d2aa39940b9414a3da1`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:49.325029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 833/1684] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:49 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:49.330744 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-289315f3-d6bb-46e7-8e88-bd799014821a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:49.507292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 852/1685] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:49 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:49.508414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15eeac03-50e9-4238-8f64-e3129f80933e service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:49.528485 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15eeac03-50e9-4238-8f64-e3129f80933e service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:49.661809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c49443-b108-4bb6-95d7-d861b827cba6 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:49.662088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c49443-b108-4bb6-95d7-d861b827cba6 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:49.662368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c49443-b108-4bb6-95d7-d861b827cba6 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:49.664486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 853/1686] 10.222.0.26 () {54 vars in 912 bytes} [Wed Sep 28 12:14:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:14:49.674655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b341f0-6732-4526-8691-1ec492a04ee4 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:49.674965 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b341f0-6732-4526-8691-1ec492a04ee4 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:49.675277 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b341f0-6732-4526-8691-1ec492a04ee4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:49.722208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-66b341f0-6732-4526-8691-1ec492a04ee4 None None] MFA Rules not processed for user `8bf969348e7a4158acf8fb789430b0e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:49.727485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-66b341f0-6732-4526-8691-1ec492a04ee4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:49.789958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678bdaaf-ca02-4d80-a356-fcafd9ad973e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:49.790645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678bdaaf-ca02-4d80-a356-fcafd9ad973e admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:49.790858 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678bdaaf-ca02-4d80-a356-fcafd9ad973e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:49.791131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678bdaaf-ca02-4d80-a356-fcafd9ad973e admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:49.791783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678bdaaf-ca02-4d80-a356-fcafd9ad973e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:49.793751 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678bdaaf-ca02-4d80-a356-fcafd9ad973e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:49.887465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 834/1687] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:14:49 2022] POST /identity/v3/projects => generated 377 bytes in 380 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:14:49.888666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678bdaaf-ca02-4d80-a356-fcafd9ad973e admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:49.909463 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:49.934074 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 854/1688] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Sep 28 12:14:49 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:14:49.946788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:49.954097 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b341f0-6732-4526-8691-1ec492a04ee4 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:49.955725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-66b341f0-6732-4526-8691-1ec492a04ee4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:50.021547 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678bdaaf-ca02-4d80-a356-fcafd9ad973e admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:50.044483 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa12691-710b-4a30-a2e3-c3ff53720543 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:50.045984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa12691-710b-4a30-a2e3-c3ff53720543 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:50.048501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa12691-710b-4a30-a2e3-c3ff53720543 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:50.049018 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa12691-710b-4a30-a2e3-c3ff53720543 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:50.052664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa12691-710b-4a30-a2e3-c3ff53720543 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:50.058155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa12691-710b-4a30-a2e3-c3ff53720543 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:50.115789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 835/1689] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:14:49 2022] POST /identity/v3/users => generated 295 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:14:50.117287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa12691-710b-4a30-a2e3-c3ff53720543 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:50.140112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa12691-710b-4a30-a2e3-c3ff53720543 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:50.166773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e7c854-e6ee-460c-9aa8-4f948307c4f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:50.169501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e7c854-e6ee-460c-9aa8-4f948307c4f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:50.173905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e7c854-e6ee-460c-9aa8-4f948307c4f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:50.174477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e7c854-e6ee-460c-9aa8-4f948307c4f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:50.175425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e7c854-e6ee-460c-9aa8-4f948307c4f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:50.177436 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e7c854-e6ee-460c-9aa8-4f948307c4f9 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:50.199128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b341f0-6732-4526-8691-1ec492a04ee4 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:50.204269 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 836/1690] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:50 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:50.211238 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e7c854-e6ee-460c-9aa8-4f948307c4f9 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:50.238473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e7c854-e6ee-460c-9aa8-4f948307c4f9 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:50.251522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e1b246-d328-46cc-af76-57604e0356d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:50.252737 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e1b246-d328-46cc-af76-57604e0356d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:50.253350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e1b246-d328-46cc-af76-57604e0356d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:50.255171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e1b246-d328-46cc-af76-57604e0356d6 admin admin] PATH_INFO: `/v3/projects/dfa48c05bb544e3495431dc2d6c016d0/users/5da1a120b31f406894d206a959148ed0/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:50.292039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e1b246-d328-46cc-af76-57604e0356d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dfa48c05bb544e3495431dc2d6c016d0, user_id=5da1a120b31f406894d206a959148ed0, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:50.297758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e1b246-d328-46cc-af76-57604e0356d6 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:50.334590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 837/1691] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:50 2022] PUT /identity/v3/projects/dfa48c05bb544e3495431dc2d6c016d0/users/5da1a120b31f406894d206a959148ed0/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:50.335913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e1b246-d328-46cc-af76-57604e0356d6 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:50.389235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83904c17-e553-4597-b956-5d50b35167ab service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '815504693bcc40b6aef06aacb2a9a35b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:50.389941 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83904c17-e553-4597-b956-5d50b35167ab service glance] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:50.390228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83904c17-e553-4597-b956-5d50b35167ab service glance] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:50.390503 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83904c17-e553-4597-b956-5d50b35167ab service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:50.505619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e1b246-d328-46cc-af76-57604e0356d6 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:50.522771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9133ee82-dc7b-45c6-8e46-626a177a89dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:50.523763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9133ee82-dc7b-45c6-8e46-626a177a89dd admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:50.524256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9133ee82-dc7b-45c6-8e46-626a177a89dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:50.524761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9133ee82-dc7b-45c6-8e46-626a177a89dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:50.525658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9133ee82-dc7b-45c6-8e46-626a177a89dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:50.528373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9133ee82-dc7b-45c6-8e46-626a177a89dd admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:50.558707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 838/1692] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:14:50 2022] GET /identity/v3/roles => generated 1420 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:14:50.559902 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9133ee82-dc7b-45c6-8e46-626a177a89dd admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:50.577791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83904c17-e553-4597-b956-5d50b35167ab service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:50.580145 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83904c17-e553-4597-b956-5d50b35167ab service glance] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:50.583303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9133ee82-dc7b-45c6-8e46-626a177a89dd admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:50.597085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15551c39-8816-4fc2-a608-5b2aa2130099 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:50.598864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15551c39-8816-4fc2-a608-5b2aa2130099 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:50.598864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15551c39-8816-4fc2-a608-5b2aa2130099 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:50.599066 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15551c39-8816-4fc2-a608-5b2aa2130099 admin admin] PATH_INFO: `/v3/projects/dfa48c05bb544e3495431dc2d6c016d0/users/5da1a120b31f406894d206a959148ed0/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:50.602872 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15551c39-8816-4fc2-a608-5b2aa2130099 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dfa48c05bb544e3495431dc2d6c016d0, user_id=5da1a120b31f406894d206a959148ed0, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:50.605892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15551c39-8816-4fc2-a608-5b2aa2130099 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:50.606906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 855/1693] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 655 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:14:50.629975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 839/1694] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:14:50 2022] PUT /identity/v3/projects/dfa48c05bb544e3495431dc2d6c016d0/users/5da1a120b31f406894d206a959148ed0/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:50.631602 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86fd05b-1dc9-4e5e-a872-b4ec76ad8a17 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:50.632481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86fd05b-1dc9-4e5e-a872-b4ec76ad8a17 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:50.632807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86fd05b-1dc9-4e5e-a872-b4ec76ad8a17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:50.668344 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-f86fd05b-1dc9-4e5e-a872-b4ec76ad8a17 None None] MFA Rules not processed for user `5da1a120b31f406894d206a959148ed0`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:50.669795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f86fd05b-1dc9-4e5e-a872-b4ec76ad8a17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:50.723414 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:50.795924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 856/1695] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:50 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:50.808471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15551c39-8816-4fc2-a608-5b2aa2130099 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:50.949374 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15551c39-8816-4fc2-a608-5b2aa2130099 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:50.961347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bedf80-3011-43ea-ba0e-67079d4d148b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:50.964972 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bedf80-3011-43ea-ba0e-67079d4d148b service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:50.965148 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bedf80-3011-43ea-ba0e-67079d4d148b service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:50.965428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bedf80-3011-43ea-ba0e-67079d4d148b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:51.013561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:51.102121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bedf80-3011-43ea-ba0e-67079d4d148b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:51.103602 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bedf80-3011-43ea-ba0e-67079d4d148b service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:51.132485 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 840/1696] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:50 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:51.480364 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ebd04-a0b0-4e5a-bda5-31dba2e16099 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:51.480364 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ebd04-a0b0-4e5a-bda5-31dba2e16099 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:51.480986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ebd04-a0b0-4e5a-bda5-31dba2e16099 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:51.482412 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 857/1697] 10.222.0.26 () {54 vars in 912 bytes} [Wed Sep 28 12:14:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:14:51.501161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676da323-fb89-4878-8dd8-2202db5a106a None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:51.501763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676da323-fb89-4878-8dd8-2202db5a106a None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:51.502828 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676da323-fb89-4878-8dd8-2202db5a106a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:51.631960 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-676da323-fb89-4878-8dd8-2202db5a106a None None] MFA Rules not processed for user `0e429f061bd04c28b0c60c593b64fb8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:51.633284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-676da323-fb89-4878-8dd8-2202db5a106a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:51.840849 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 841/1698] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Sep 28 12:14:51 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 345 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:51.903734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21ebd04-a0b0-4e5a-bda5-31dba2e16099 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:51.906711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e21ebd04-a0b0-4e5a-bda5-31dba2e16099 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:51.943302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21ebd04-a0b0-4e5a-bda5-31dba2e16099 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:51.955453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8cbbb0-ccf9-404d-998f-9cc74fa99f7c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:51.958788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8cbbb0-ccf9-404d-998f-9cc74fa99f7c service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:51.959072 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8cbbb0-ccf9-404d-998f-9cc74fa99f7c service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:51.961516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8cbbb0-ccf9-404d-998f-9cc74fa99f7c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:51.989552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8cbbb0-ccf9-404d-998f-9cc74fa99f7c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:51.991079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8cbbb0-ccf9-404d-998f-9cc74fa99f7c service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:52.009567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 858/1699] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:51 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:53.200181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676da323-fb89-4878-8dd8-2202db5a106a None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:53.202787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-676da323-fb89-4878-8dd8-2202db5a106a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:53.399715 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676da323-fb89-4878-8dd8-2202db5a106a None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:53.413727 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad68884-1391-4c5f-a382-6da0dc8c091c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:53.416778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad68884-1391-4c5f-a382-6da0dc8c091c service placement] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:53.417683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad68884-1391-4c5f-a382-6da0dc8c091c service placement] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:53.417966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad68884-1391-4c5f-a382-6da0dc8c091c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:53.423481 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2ad68884-1391-4c5f-a382-6da0dc8c091c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:53.454191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad68884-1391-4c5f-a382-6da0dc8c091c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:53.458486 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad68884-1391-4c5f-a382-6da0dc8c091c service placement] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:53.493548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 842/1700] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:53.868898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab17eb5f-a95b-4144-a994-3a2a446a8f2e None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:53.869617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab17eb5f-a95b-4144-a994-3a2a446a8f2e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:53.870044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab17eb5f-a95b-4144-a994-3a2a446a8f2e None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:53.873046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 859/1701] 10.222.0.26 () {54 vars in 961 bytes} [Wed Sep 28 12:14:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:14:53.890054 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d285d1d5-f4bc-405b-b0f1-bbca448fccb8 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:53.890694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d285d1d5-f4bc-405b-b0f1-bbca448fccb8 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:53.891199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d285d1d5-f4bc-405b-b0f1-bbca448fccb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:53.953682 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-d285d1d5-f4bc-405b-b0f1-bbca448fccb8 None None] MFA Rules not processed for user `8bf969348e7a4158acf8fb789430b0e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:53.957590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d285d1d5-f4bc-405b-b0f1-bbca448fccb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:54.171542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:54.289661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 843/1702] 10.222.0.26 () {58 vars in 1058 bytes} [Wed Sep 28 12:14:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 400 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:54.296802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9657fc-3fc2-4997-bcd1-e0cd192832a9 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:54.297415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9657fc-3fc2-4997-bcd1-e0cd192832a9 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:54.297826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9657fc-3fc2-4997-bcd1-e0cd192832a9 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:54.299334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 860/1703] 10.222.0.26 () {54 vars in 912 bytes} [Wed Sep 28 12:14:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:14:54.308031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bff60b7-181f-4b29-ae45-90a072e2edc7 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:54.308583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bff60b7-181f-4b29-ae45-90a072e2edc7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:54.308997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bff60b7-181f-4b29-ae45-90a072e2edc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:54.370186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-1bff60b7-181f-4b29-ae45-90a072e2edc7 None None] MFA Rules not processed for user `8bf969348e7a4158acf8fb789430b0e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:54.376972 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1bff60b7-181f-4b29-ae45-90a072e2edc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:54.416595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 844/1704] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Sep 28 12:14:54 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:54.829873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9657fc-3fc2-4997-bcd1-e0cd192832a9 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:54.832147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-dc9657fc-3fc2-4997-bcd1-e0cd192832a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:54.835803 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3099edd7-70fb-49d8-b4c1-53eaa662a566 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:54.836252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3099edd7-70fb-49d8-b4c1-53eaa662a566 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:54.837396 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3099edd7-70fb-49d8-b4c1-53eaa662a566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:54.876558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9657fc-3fc2-4997-bcd1-e0cd192832a9 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:54.885283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63de7548-a5ef-4684-9aed-02f0606d56f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:54.890765 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63de7548-a5ef-4684-9aed-02f0606d56f2 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:54.892432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63de7548-a5ef-4684-9aed-02f0606d56f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:54.892719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63de7548-a5ef-4684-9aed-02f0606d56f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:54.921008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3099edd7-70fb-49d8-b4c1-53eaa662a566 None None] MFA Rules not processed for user `fdb0f345d2e04483b5166e1eb728bf91`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:54.928115 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63de7548-a5ef-4684-9aed-02f0606d56f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:54.928744 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3099edd7-70fb-49d8-b4c1-53eaa662a566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:54.931812 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63de7548-a5ef-4684-9aed-02f0606d56f2 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:54.968407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 861/1705] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:55.130380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63de7548-a5ef-4684-9aed-02f0606d56f2 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:55.167385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 845/1706] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:54 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 332 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:55.184510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3099edd7-70fb-49d8-b4c1-53eaa662a566 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:55.207815 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3099edd7-70fb-49d8-b4c1-53eaa662a566 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:55.239687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c910a60b-53e5-462a-bdbf-37b5acad145e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:55.240346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c910a60b-53e5-462a-bdbf-37b5acad145e service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:55.240908 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c910a60b-53e5-462a-bdbf-37b5acad145e service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:55.241414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c910a60b-53e5-462a-bdbf-37b5acad145e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:55.273429 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c910a60b-53e5-462a-bdbf-37b5acad145e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:55.277544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c910a60b-53e5-462a-bdbf-37b5acad145e service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:55.299412 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63de7548-a5ef-4684-9aed-02f0606d56f2 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:55.307811 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a3f6bb-4ee4-41e2-8473-e30f826e141e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:55.308658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 846/1707] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:55 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:14:55.309140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a3f6bb-4ee4-41e2-8473-e30f826e141e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:55.309140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a3f6bb-4ee4-41e2-8473-e30f826e141e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:55.309613 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a3f6bb-4ee4-41e2-8473-e30f826e141e admin admin] PATH_INFO: `/v3/users/6ecfc15fffa044f68a092a88b9c3dfeb` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:55.312264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a3f6bb-4ee4-41e2-8473-e30f826e141e admin admin] RBAC: Authorizing `identity:delete_user(user_id=6ecfc15fffa044f68a092a88b9c3dfeb)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:55.321043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a3f6bb-4ee4-41e2-8473-e30f826e141e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:55.459345 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-88a3f6bb-4ee4-41e2-8473-e30f826e141e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6ecfc15fffa044f68a092a88b9c3dfeb'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:55.461151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-88a3f6bb-4ee4-41e2-8473-e30f826e141e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6ecfc15fffa044f68a092a88b9c3dfeb'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:55.487965 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-88a3f6bb-4ee4-41e2-8473-e30f826e141e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6ecfc15fffa044f68a092a88b9c3dfeb'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:55.504632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-88a3f6bb-4ee4-41e2-8473-e30f826e141e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6ecfc15fffa044f68a092a88b9c3dfeb'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:55.520776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea6b6e3-1399-4f5f-97fc-eaff6fbade32 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:55.521473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea6b6e3-1399-4f5f-97fc-eaff6fbade32 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:55.521903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea6b6e3-1399-4f5f-97fc-eaff6fbade32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:55.537456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-88a3f6bb-4ee4-41e2-8473-e30f826e141e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6ecfc15fffa044f68a092a88b9c3dfeb'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:55.565731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3ea6b6e3-1399-4f5f-97fc-eaff6fbade32 None None] MFA Rules not processed for user `956c7f92a4a747dba49e3f3a527bba74`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:55.567351 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3ea6b6e3-1399-4f5f-97fc-eaff6fbade32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:55.569725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 862/1708] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:14:55 2022] DELETE /identity/v3/users/6ecfc15fffa044f68a092a88b9c3dfeb => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:14:55.573862 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53af1a6b-7fb7-4ba8-b750-ef585965f8fa None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:55.574169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53af1a6b-7fb7-4ba8-b750-ef585965f8fa None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:55.574432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53af1a6b-7fb7-4ba8-b750-ef585965f8fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:55.608547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-53af1a6b-7fb7-4ba8-b750-ef585965f8fa None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:55.610006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-53af1a6b-7fb7-4ba8-b750-ef585965f8fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:55.631625 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:55.729413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 847/1709] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:55 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:55.767315 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 863/1710] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:14:55.797783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea6b6e3-1399-4f5f-97fc-eaff6fbade32 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:55.799336 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3ea6b6e3-1399-4f5f-97fc-eaff6fbade32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:56.046766 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea6b6e3-1399-4f5f-97fc-eaff6fbade32 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:56.051962 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0d9fcf-f73d-43d2-b9ff-0ee9a717d94d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:56.052507 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0d9fcf-f73d-43d2-b9ff-0ee9a717d94d service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:56.052897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0d9fcf-f73d-43d2-b9ff-0ee9a717d94d service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:56.053187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0d9fcf-f73d-43d2-b9ff-0ee9a717d94d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:56.092983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:56.097010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0d9fcf-f73d-43d2-b9ff-0ee9a717d94d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:56.105725 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0d9fcf-f73d-43d2-b9ff-0ee9a717d94d service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:56.148755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 848/1711] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:56.252185 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c368d8-27bf-4dda-a70d-2bb5449468f1 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:56.253149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c368d8-27bf-4dda-a70d-2bb5449468f1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:56.253595 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c368d8-27bf-4dda-a70d-2bb5449468f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:56.322427 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-58c368d8-27bf-4dda-a70d-2bb5449468f1 None None] MFA Rules not processed for user `3b0e4c6131674b3799a374aacdec708f`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:56.326443 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-58c368d8-27bf-4dda-a70d-2bb5449468f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:56.507777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 864/1712] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:56 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:56.521620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69064d9-3ddc-4ba3-80be-161cb8145aa2 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:56.522151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69064d9-3ddc-4ba3-80be-161cb8145aa2 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:56.522429 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69064d9-3ddc-4ba3-80be-161cb8145aa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:56.571056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-a69064d9-3ddc-4ba3-80be-161cb8145aa2 None None] MFA Rules not processed for user `3b0e4c6131674b3799a374aacdec708f`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:56.572511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a69064d9-3ddc-4ba3-80be-161cb8145aa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:56.603583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 849/1713] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:56 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:56.611778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c368d8-27bf-4dda-a70d-2bb5449468f1 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:56.613458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-58c368d8-27bf-4dda-a70d-2bb5449468f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:56.771680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c368d8-27bf-4dda-a70d-2bb5449468f1 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:56.788429 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d207691-79f1-4ceb-922e-932a5cb8643c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:56.789173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d207691-79f1-4ceb-922e-932a5cb8643c service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:56.789544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d207691-79f1-4ceb-922e-932a5cb8643c service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:56.789893 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d207691-79f1-4ceb-922e-932a5cb8643c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:56.806489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d207691-79f1-4ceb-922e-932a5cb8643c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:56.808197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d207691-79f1-4ceb-922e-932a5cb8643c service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:56.840064 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 865/1714] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:56 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:57.421966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b751dce-7f58-4049-b210-9046b701b58f None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:57.422255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b751dce-7f58-4049-b210-9046b701b58f None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:57.422532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b751dce-7f58-4049-b210-9046b701b58f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:57.446192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-5b751dce-7f58-4049-b210-9046b701b58f None None] MFA Rules not processed for user `0e429f061bd04c28b0c60c593b64fb8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:57.447867 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5b751dce-7f58-4049-b210-9046b701b58f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:57.464732 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 850/1715] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Sep 28 12:14:57 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:14:57.494631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d207691-79f1-4ceb-922e-932a5cb8643c service nova] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:57.527891 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d207691-79f1-4ceb-922e-932a5cb8643c service nova] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:57.539895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7300aeb-922e-4098-97ac-a3d8eaa493cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:57.540554 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7300aeb-922e-4098-97ac-a3d8eaa493cc service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:57.540833 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7300aeb-922e-4098-97ac-a3d8eaa493cc service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:57.541107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7300aeb-922e-4098-97ac-a3d8eaa493cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:57.563472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7300aeb-922e-4098-97ac-a3d8eaa493cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:57.565168 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7300aeb-922e-4098-97ac-a3d8eaa493cc service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:57.589267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 866/1716] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:57 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:57.707161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb15059-d916-451c-adad-a966f57bbe0a None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:57.707702 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb15059-d916-451c-adad-a966f57bbe0a None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:57.708857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb15059-d916-451c-adad-a966f57bbe0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:57.744048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-4fb15059-d916-451c-adad-a966f57bbe0a None None] MFA Rules not processed for user `4a8818fd59954480bcff68b9ba52769a`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:14:57.746561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-4fb15059-d916-451c-adad-a966f57bbe0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:57.887203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 851/1717] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:14:58.482486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7300aeb-922e-4098-97ac-a3d8eaa493cc service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:58.485010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f7300aeb-922e-4098-97ac-a3d8eaa493cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:58.537413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7300aeb-922e-4098-97ac-a3d8eaa493cc service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:58.555567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec7f94d-99af-4366-a34d-a4c4e26b1377 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:58.557174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec7f94d-99af-4366-a34d-a4c4e26b1377 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:58.557462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec7f94d-99af-4366-a34d-a4c4e26b1377 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:58.557732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec7f94d-99af-4366-a34d-a4c4e26b1377 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:58.595289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec7f94d-99af-4366-a34d-a4c4e26b1377 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:58.596951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec7f94d-99af-4366-a34d-a4c4e26b1377 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:58.633895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 867/1718] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:14:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:14:58.975574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb15059-d916-451c-adad-a966f57bbe0a None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:58.984927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-4fb15059-d916-451c-adad-a966f57bbe0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:14:59.037944 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb15059-d916-451c-adad-a966f57bbe0a None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:59.053046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:59.053895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:59.056407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:59.057477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641 admin admin] PATH_INFO: `/v3/projects/8570c618753b487d8a0f9e1a7c5348f4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:59.077984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8570c618753b487d8a0f9e1a7c5348f4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:59.080035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:59.205636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8570c618753b487d8a0f9e1a7c5348f4', 'request_id': 'req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:59.244785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8570c618753b487d8a0f9e1a7c5348f4', 'request_id': 'req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:59.267987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8570c618753b487d8a0f9e1a7c5348f4', 'request_id': 'req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:59.308517 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641 admin admin] The token cache is being invalidate because project 8570c618753b487d8a0f9e1a7c5348f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:14:59.310551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:59.312162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-207ed022-bc0f-4313-bdc2-a1c9a6a2a641 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:59.314947 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 852/1719] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:14:58 2022] DELETE /identity/v3/projects/8570c618753b487d8a0f9e1a7c5348f4 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:59.320997 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec7f94d-99af-4366-a34d-a4c4e26b1377 service nova] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:14:59.331626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:14:59.349650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec7f94d-99af-4366-a34d-a4c4e26b1377 service nova] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:14:59.603066 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164bf807-44db-4210-a25d-4ae50ab8507f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:14:59.609982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164bf807-44db-4210-a25d-4ae50ab8507f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:59.626369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164bf807-44db-4210-a25d-4ae50ab8507f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:59.627021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164bf807-44db-4210-a25d-4ae50ab8507f admin admin] PATH_INFO: `/v3/users/cafd469d08d0431586c3ef4d273f9b46` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:59.632849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164bf807-44db-4210-a25d-4ae50ab8507f admin admin] RBAC: Authorizing `identity:delete_user(user_id=cafd469d08d0431586c3ef4d273f9b46)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:14:59.663455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164bf807-44db-4210-a25d-4ae50ab8507f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:14:59.813924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-164bf807-44db-4210-a25d-4ae50ab8507f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cafd469d08d0431586c3ef4d273f9b46'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:59.816059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-164bf807-44db-4210-a25d-4ae50ab8507f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cafd469d08d0431586c3ef4d273f9b46'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:59.852093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-164bf807-44db-4210-a25d-4ae50ab8507f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cafd469d08d0431586c3ef4d273f9b46'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:59.852999 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a786477-0f44-4fa7-b9e5-38281aab75b7 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:14:59.853120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-164bf807-44db-4210-a25d-4ae50ab8507f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cafd469d08d0431586c3ef4d273f9b46'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:59.853287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a786477-0f44-4fa7-b9e5-38281aab75b7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:14:59.853562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a786477-0f44-4fa7-b9e5-38281aab75b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:14:59.892563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-164bf807-44db-4210-a25d-4ae50ab8507f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cafd469d08d0431586c3ef4d273f9b46'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:14:59.929693 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 868/1720] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:14:59 2022] DELETE /identity/v3/users/cafd469d08d0431586c3ef4d273f9b46 => generated 0 bytes in 611 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:14:59.998014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-6a786477-0f44-4fa7-b9e5-38281aab75b7 None None] MFA Rules not processed for user `820f4bb963644499a77fa1f3edd59d08`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:00.006386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6a786477-0f44-4fa7-b9e5-38281aab75b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:00.055892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164bf807-44db-4210-a25d-4ae50ab8507f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:00.057710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-164bf807-44db-4210-a25d-4ae50ab8507f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:00.244490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164bf807-44db-4210-a25d-4ae50ab8507f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:00.260026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d754fa7-a1cf-44ec-aef1-9e12706b88c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:00.266916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d754fa7-a1cf-44ec-aef1-9e12706b88c5 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:00.268831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d754fa7-a1cf-44ec-aef1-9e12706b88c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:00.268831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d754fa7-a1cf-44ec-aef1-9e12706b88c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:00.283766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6d754fa7-a1cf-44ec-aef1-9e12706b88c5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:00.358513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:00.445121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:00.545856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 853/1721] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:14:59 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 693 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:00.554694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a786477-0f44-4fa7-b9e5-38281aab75b7 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:00.561964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6a786477-0f44-4fa7-b9e5-38281aab75b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:00.620539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d754fa7-a1cf-44ec-aef1-9e12706b88c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:00.626966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d754fa7-a1cf-44ec-aef1-9e12706b88c5 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:00.668453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 869/1722] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:00 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 613 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:15:00.883142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a786477-0f44-4fa7-b9e5-38281aab75b7 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:00.927902 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc46b889-f291-4b03-9f00-4874e4f9dfc8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:00.930851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc46b889-f291-4b03-9f00-4874e4f9dfc8 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:00.931610 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc46b889-f291-4b03-9f00-4874e4f9dfc8 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:00.934746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc46b889-f291-4b03-9f00-4874e4f9dfc8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:00.965354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc46b889-f291-4b03-9f00-4874e4f9dfc8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:00.967362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc46b889-f291-4b03-9f00-4874e4f9dfc8 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:01.051405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 854/1723] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:00 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 494 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:01.163894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ba21d8-a3a0-4fbf-8828-febc5af01397 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:01.164343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ba21d8-a3a0-4fbf-8828-febc5af01397 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:01.164616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ba21d8-a3a0-4fbf-8828-febc5af01397 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:01.316983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-17ba21d8-a3a0-4fbf-8828-febc5af01397 None None] MFA Rules not processed for user `b20a6fb4e7a546058f5fa314c9a59299`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:01.324031 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-17ba21d8-a3a0-4fbf-8828-febc5af01397 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:01.698726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 870/1724] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Sep 28 12:15:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 536 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:02.024720 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc46b889-f291-4b03-9f00-4874e4f9dfc8 service nova] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:02.026529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-cc46b889-f291-4b03-9f00-4874e4f9dfc8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:02.054597 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c10f1c-c6fe-4075-b406-b6a212946cc8 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:02.054885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c10f1c-c6fe-4075-b406-b6a212946cc8 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:02.055164 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c10f1c-c6fe-4075-b406-b6a212946cc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:02.126528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-b2c10f1c-c6fe-4075-b406-b6a212946cc8 None None] MFA Rules not processed for user `fdb0f345d2e04483b5166e1eb728bf91`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:02.127858 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b2c10f1c-c6fe-4075-b406-b6a212946cc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:02.351794 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 871/1725] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:02 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 297 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:02.357374 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc46b889-f291-4b03-9f00-4874e4f9dfc8 service nova] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:02.365425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9296663a-aae1-45a7-b66e-215e9589d5c1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e663aad79bb8429aa223564a15332ee2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:02.366148 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9296663a-aae1-45a7-b66e-215e9589d5c1 service swift] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:02.367577 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9296663a-aae1-45a7-b66e-215e9589d5c1 service swift] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:02.367844 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9296663a-aae1-45a7-b66e-215e9589d5c1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:02.381392 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9296663a-aae1-45a7-b66e-215e9589d5c1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:02.383012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9296663a-aae1-45a7-b66e-215e9589d5c1 service swift] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:02.394516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 855/1726] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Sep 28 12:15:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 28 12:15:03.914485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c10f1c-c6fe-4075-b406-b6a212946cc8 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:03.917957 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b2c10f1c-c6fe-4075-b406-b6a212946cc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:04.052859 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:04.092239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c10f1c-c6fe-4075-b406-b6a212946cc8 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:04.116389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d048f130-7ed6-4f52-b77a-5d15a3844a06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:04.117089 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d048f130-7ed6-4f52-b77a-5d15a3844a06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:04.117372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d048f130-7ed6-4f52-b77a-5d15a3844a06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:04.117645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d048f130-7ed6-4f52-b77a-5d15a3844a06 admin admin] PATH_INFO: `/v3/projects/c3ff76027be148cf81f57f8d1ccb930f` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:04.119203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d048f130-7ed6-4f52-b77a-5d15a3844a06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c3ff76027be148cf81f57f8d1ccb930f)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:04.122550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d048f130-7ed6-4f52-b77a-5d15a3844a06 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:04.176218 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4e8a89-ef70-481b-860f-766148c37f83 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:04.178888 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4e8a89-ef70-481b-860f-766148c37f83 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:04.179338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4e8a89-ef70-481b-860f-766148c37f83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:04.236970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-8e4e8a89-ef70-481b-860f-766148c37f83 None None] MFA Rules not processed for user `4a8818fd59954480bcff68b9ba52769a`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:04.239989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8e4e8a89-ef70-481b-860f-766148c37f83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:04.272432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d048f130-7ed6-4f52-b77a-5d15a3844a06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c3ff76027be148cf81f57f8d1ccb930f', 'request_id': 'req-d048f130-7ed6-4f52-b77a-5d15a3844a06'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:04.280269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d048f130-7ed6-4f52-b77a-5d15a3844a06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c3ff76027be148cf81f57f8d1ccb930f', 'request_id': 'req-d048f130-7ed6-4f52-b77a-5d15a3844a06'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:04.294236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d048f130-7ed6-4f52-b77a-5d15a3844a06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c3ff76027be148cf81f57f8d1ccb930f', 'request_id': 'req-d048f130-7ed6-4f52-b77a-5d15a3844a06'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:04.310321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d048f130-7ed6-4f52-b77a-5d15a3844a06 admin admin] The token cache is being invalidate because project c3ff76027be148cf81f57f8d1ccb930f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:04.310724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d048f130-7ed6-4f52-b77a-5d15a3844a06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:04.312131 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d048f130-7ed6-4f52-b77a-5d15a3844a06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:04.314801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 872/1727] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:03 2022] DELETE /identity/v3/projects/c3ff76027be148cf81f57f8d1ccb930f => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:04.327353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af698db-c1e9-43dc-be5f-d3b7b2813dfc None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:04.327574 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af698db-c1e9-43dc-be5f-d3b7b2813dfc None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:04.327853 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af698db-c1e9-43dc-be5f-d3b7b2813dfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:04.344008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:04.366276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-5af698db-c1e9-43dc-be5f-d3b7b2813dfc None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:04.367877 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5af698db-c1e9-43dc-be5f-d3b7b2813dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:04.408950 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 856/1728] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:04 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:04.525339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 873/1729] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:04.531529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e70f87c-6696-49df-abc3-2a97e395b309 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:04.531822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e70f87c-6696-49df-abc3-2a97e395b309 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:04.532193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e70f87c-6696-49df-abc3-2a97e395b309 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:04.564886 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-7e70f87c-6696-49df-abc3-2a97e395b309 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:04.566210 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-7e70f87c-6696-49df-abc3-2a97e395b309 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:04.602880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 857/1730] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:04.605248 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af698db-c1e9-43dc-be5f-d3b7b2813dfc None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:04.626255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af698db-c1e9-43dc-be5f-d3b7b2813dfc None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:04.643168 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f12ee5-612a-4d57-951d-ec83876d3f9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:04.644873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f12ee5-612a-4d57-951d-ec83876d3f9b admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:04.645231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f12ee5-612a-4d57-951d-ec83876d3f9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:04.645426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f12ee5-612a-4d57-951d-ec83876d3f9b admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:04.646712 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f12ee5-612a-4d57-951d-ec83876d3f9b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:15:04.647132 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f12ee5-612a-4d57-951d-ec83876d3f9b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:04.648993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f12ee5-612a-4d57-951d-ec83876d3f9b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:04.665400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 874/1731] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:15:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:15:04.670399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e70f87c-6696-49df-abc3-2a97e395b309 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:04.687681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e70f87c-6696-49df-abc3-2a97e395b309 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:04.693562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d19d17-a8d0-43d9-bd19-0668b8b03bda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:04.694410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d19d17-a8d0-43d9-bd19-0668b8b03bda admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:04.694700 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d19d17-a8d0-43d9-bd19-0668b8b03bda admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:04.694973 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d19d17-a8d0-43d9-bd19-0668b8b03bda admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:04.695962 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d19d17-a8d0-43d9-bd19-0668b8b03bda admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:04.697629 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d19d17-a8d0-43d9-bd19-0668b8b03bda admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:04.761309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f12ee5-612a-4d57-951d-ec83876d3f9b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:04.762181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 858/1732] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:04 2022] POST /identity/v3/projects => generated 385 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:04.781861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f12ee5-612a-4d57-951d-ec83876d3f9b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:04.925018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2e76ea-c983-40c6-8afa-43a12694d2cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:04.925673 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2e76ea-c983-40c6-8afa-43a12694d2cc admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:04.926057 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2e76ea-c983-40c6-8afa-43a12694d2cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:04.926519 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2e76ea-c983-40c6-8afa-43a12694d2cc admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:04.927231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2e76ea-c983-40c6-8afa-43a12694d2cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:04.929254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2e76ea-c983-40c6-8afa-43a12694d2cc admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:04.959624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:04.973876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 875/1733] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:04 2022] POST /identity/v3/users => generated 293 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:04.977311 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d19d17-a8d0-43d9-bd19-0668b8b03bda admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:04.987882 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d19d17-a8d0-43d9-bd19-0668b8b03bda admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:04.992837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f558bf-2d7e-473a-ba67-2197207f85c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:04.993461 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f558bf-2d7e-473a-ba67-2197207f85c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:04.993736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f558bf-2d7e-473a-ba67-2197207f85c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:04.994058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f558bf-2d7e-473a-ba67-2197207f85c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:04.995714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f558bf-2d7e-473a-ba67-2197207f85c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:04.996930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f558bf-2d7e-473a-ba67-2197207f85c2 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:05.015653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 859/1734] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:04 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:05.020093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2e76ea-c983-40c6-8afa-43a12694d2cc admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:05.038430 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2e76ea-c983-40c6-8afa-43a12694d2cc admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:05.043293 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af0151f-309e-4409-bd43-14a6c320ceb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:05.043918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af0151f-309e-4409-bd43-14a6c320ceb3 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:05.044194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af0151f-309e-4409-bd43-14a6c320ceb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:05.044494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af0151f-309e-4409-bd43-14a6c320ceb3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:05.045579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af0151f-309e-4409-bd43-14a6c320ceb3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:05.046833 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af0151f-309e-4409-bd43-14a6c320ceb3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:05.067707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 876/1735] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:05 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:05.071653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f558bf-2d7e-473a-ba67-2197207f85c2 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:05.085959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f558bf-2d7e-473a-ba67-2197207f85c2 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:05.098898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4819f5-cd4a-4ef0-85c3-fc7cec9f231c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:05.099254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4819f5-cd4a-4ef0-85c3-fc7cec9f231c admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:05.099514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4819f5-cd4a-4ef0-85c3-fc7cec9f231c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:05.099784 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4819f5-cd4a-4ef0-85c3-fc7cec9f231c admin admin] PATH_INFO: `/v3/projects/5635233bf7394bd199c675fefc0fe603/users/3bb8301f69c54c4ebf6f07863f767cd0/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:05.121023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4819f5-cd4a-4ef0-85c3-fc7cec9f231c admin admin] RBAC: Authorizing `identity:create_grant(project_id=5635233bf7394bd199c675fefc0fe603, user_id=3bb8301f69c54c4ebf6f07863f767cd0, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:05.124717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4819f5-cd4a-4ef0-85c3-fc7cec9f231c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:05.143688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 860/1736] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:05 2022] PUT /identity/v3/projects/5635233bf7394bd199c675fefc0fe603/users/3bb8301f69c54c4ebf6f07863f767cd0/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:05.154857 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9da10f2-7746-4fb1-8592-23d18b7d3f3b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:05.155817 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9da10f2-7746-4fb1-8592-23d18b7d3f3b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:05.156096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9da10f2-7746-4fb1-8592-23d18b7d3f3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:05.210253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-b9da10f2-7746-4fb1-8592-23d18b7d3f3b None None] MFA Rules not processed for user `3bb8301f69c54c4ebf6f07863f767cd0`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:05.211544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b9da10f2-7746-4fb1-8592-23d18b7d3f3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:05.312093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 877/1737] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:05 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:05.314101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4819f5-cd4a-4ef0-85c3-fc7cec9f231c admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:05.469077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4819f5-cd4a-4ef0-85c3-fc7cec9f231c admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:05.497632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bca1027-8de5-4998-81fc-0c71f64ecc0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:05.497632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bca1027-8de5-4998-81fc-0c71f64ecc0f admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:05.497632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bca1027-8de5-4998-81fc-0c71f64ecc0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:05.497632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bca1027-8de5-4998-81fc-0c71f64ecc0f admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:05.498231 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bca1027-8de5-4998-81fc-0c71f64ecc0f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:05.500603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bca1027-8de5-4998-81fc-0c71f64ecc0f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:05.574952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 861/1738] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:05 2022] POST /identity/v3/projects => generated 387 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:05.576076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9da10f2-7746-4fb1-8592-23d18b7d3f3b None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:05.597811 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9da10f2-7746-4fb1-8592-23d18b7d3f3b None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:05.763174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5763681a-7128-4849-aa0f-d2eab67d9d4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:05.765510 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5763681a-7128-4849-aa0f-d2eab67d9d4c admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:05.765637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5763681a-7128-4849-aa0f-d2eab67d9d4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:05.766314 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5763681a-7128-4849-aa0f-d2eab67d9d4c admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:05.767664 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5763681a-7128-4849-aa0f-d2eab67d9d4c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:05.773201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5763681a-7128-4849-aa0f-d2eab67d9d4c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:05.845070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bca1027-8de5-4998-81fc-0c71f64ecc0f admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:05.846495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 878/1739] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:05 2022] POST /identity/v3/users => generated 300 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:05.869508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bca1027-8de5-4998-81fc-0c71f64ecc0f admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:05.882001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd51ab50-45b6-41d4-a8be-a038c72cbe12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:05.882640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51ab50-45b6-41d4-a8be-a038c72cbe12 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:05.882978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51ab50-45b6-41d4-a8be-a038c72cbe12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:05.883173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51ab50-45b6-41d4-a8be-a038c72cbe12 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:05.893354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd51ab50-45b6-41d4-a8be-a038c72cbe12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:05.902254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd51ab50-45b6-41d4-a8be-a038c72cbe12 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:05.972457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 862/1740] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:05 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:05.978155 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5763681a-7128-4849-aa0f-d2eab67d9d4c admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:05.999562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5763681a-7128-4849-aa0f-d2eab67d9d4c admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:06.024555 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8761f9-9a4b-4b1d-93e9-3b4588a7f24d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:06.026635 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8761f9-9a4b-4b1d-93e9-3b4588a7f24d admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:06.028617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8761f9-9a4b-4b1d-93e9-3b4588a7f24d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:06.029119 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8761f9-9a4b-4b1d-93e9-3b4588a7f24d admin admin] PATH_INFO: `/v3/projects/ed9302581e5b4d3b8e043f83cbd3aef9/users/6554334bf26f4048b8a407be0eb3b73a/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:06.088248 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8761f9-9a4b-4b1d-93e9-3b4588a7f24d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed9302581e5b4d3b8e043f83cbd3aef9, user_id=6554334bf26f4048b8a407be0eb3b73a, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:06.095041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8761f9-9a4b-4b1d-93e9-3b4588a7f24d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:06.147933 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 879/1741] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:05 2022] PUT /identity/v3/projects/ed9302581e5b4d3b8e043f83cbd3aef9/users/6554334bf26f4048b8a407be0eb3b73a/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:06.150495 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd51ab50-45b6-41d4-a8be-a038c72cbe12 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:06.173681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd51ab50-45b6-41d4-a8be-a038c72cbe12 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:06.304737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1dd1e3-d4e2-4da3-ba4a-f78613c35078 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:06.305407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1dd1e3-d4e2-4da3-ba4a-f78613c35078 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:06.305831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1dd1e3-d4e2-4da3-ba4a-f78613c35078 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:06.341537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:06.367655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-aa1dd1e3-d4e2-4da3-ba4a-f78613c35078 None None] MFA Rules not processed for user `0e429f061bd04c28b0c60c593b64fb8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:06.369007 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-aa1dd1e3-d4e2-4da3-ba4a-f78613c35078 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:06.370606 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a008eee4-3e03-4a34-bd5e-51f6ab268504 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:06.371407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a008eee4-3e03-4a34-bd5e-51f6ab268504 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:06.371822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a008eee4-3e03-4a34-bd5e-51f6ab268504 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:06.372230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a008eee4-3e03-4a34-bd5e-51f6ab268504 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:06.375541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a008eee4-3e03-4a34-bd5e-51f6ab268504 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:06.377391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a008eee4-3e03-4a34-bd5e-51f6ab268504 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:06.419349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 863/1742] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:06 2022] GET /identity/v3/roles => generated 1420 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:15:06.420862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a008eee4-3e03-4a34-bd5e-51f6ab268504 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:06.459207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a008eee4-3e03-4a34-bd5e-51f6ab268504 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:06.473827 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:06.484699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712abc3d-623b-43a8-8e22-207b3857eead admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:06.488135 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712abc3d-623b-43a8-8e22-207b3857eead admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:06.488962 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712abc3d-623b-43a8-8e22-207b3857eead admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:06.489891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712abc3d-623b-43a8-8e22-207b3857eead admin admin] PATH_INFO: `/v3/projects/ed9302581e5b4d3b8e043f83cbd3aef9/users/6554334bf26f4048b8a407be0eb3b73a/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:06.492718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712abc3d-623b-43a8-8e22-207b3857eead admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed9302581e5b4d3b8e043f83cbd3aef9, user_id=6554334bf26f4048b8a407be0eb3b73a, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:06.502671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712abc3d-623b-43a8-8e22-207b3857eead admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:06.512579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 880/1743] 10.222.0.25 () {58 vars in 1015 bytes} [Wed Sep 28 12:15:06 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:06.539786 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 864/1744] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:06 2022] PUT /identity/v3/projects/ed9302581e5b4d3b8e043f83cbd3aef9/users/6554334bf26f4048b8a407be0eb3b73a/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:06.554373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efa61de-ff58-416f-aa28-2e4ceb4413e5 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:06.555313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efa61de-ff58-416f-aa28-2e4ceb4413e5 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:06.555313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efa61de-ff58-416f-aa28-2e4ceb4413e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:06.566148 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712abc3d-623b-43a8-8e22-207b3857eead admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:06.567796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-712abc3d-623b-43a8-8e22-207b3857eead admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:06.659689 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-5efa61de-ff58-416f-aa28-2e4ceb4413e5 None None] MFA Rules not processed for user `6554334bf26f4048b8a407be0eb3b73a`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:06.660989 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5efa61de-ff58-416f-aa28-2e4ceb4413e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:06.732795 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712abc3d-623b-43a8-8e22-207b3857eead admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:06.747048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a073b18-d5d6-4514-b64a-de5e9c780609 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:06.747693 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a073b18-d5d6-4514-b64a-de5e9c780609 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:06.747976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a073b18-d5d6-4514-b64a-de5e9c780609 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:06.751129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a073b18-d5d6-4514-b64a-de5e9c780609 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:06.800387 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a073b18-d5d6-4514-b64a-de5e9c780609 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:06.802393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a073b18-d5d6-4514-b64a-de5e9c780609 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:06.839580 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 865/1745] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 274 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:15:06.864446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 881/1746] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:06 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:06.874962 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a073b18-d5d6-4514-b64a-de5e9c780609 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:06.877112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9a073b18-d5d6-4514-b64a-de5e9c780609 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:07.019556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a073b18-d5d6-4514-b64a-de5e9c780609 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:07.033029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8fd471-0acd-4648-915d-6a9c07101dd4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:07.034060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8fd471-0acd-4648-915d-6a9c07101dd4 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:07.034355 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8fd471-0acd-4648-915d-6a9c07101dd4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:07.034667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8fd471-0acd-4648-915d-6a9c07101dd4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:07.059817 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8fd471-0acd-4648-915d-6a9c07101dd4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:07.063801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8fd471-0acd-4648-915d-6a9c07101dd4 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:07.102693 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 866/1747] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:06 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:07.195317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efa61de-ff58-416f-aa28-2e4ceb4413e5 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:07.457701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efa61de-ff58-416f-aa28-2e4ceb4413e5 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:07.488769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7118c76a-fecb-4a7c-a16e-2d490a2b56b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:07.493475 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7118c76a-fecb-4a7c-a16e-2d490a2b56b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:07.493769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7118c76a-fecb-4a7c-a16e-2d490a2b56b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:07.494035 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7118c76a-fecb-4a7c-a16e-2d490a2b56b9 admin admin] PATH_INFO: `/v3/users/3bb8301f69c54c4ebf6f07863f767cd0` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:07.504992 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7118c76a-fecb-4a7c-a16e-2d490a2b56b9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3bb8301f69c54c4ebf6f07863f767cd0)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:07.507281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7118c76a-fecb-4a7c-a16e-2d490a2b56b9 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:07.671009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7118c76a-fecb-4a7c-a16e-2d490a2b56b9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3bb8301f69c54c4ebf6f07863f767cd0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:07.673324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7118c76a-fecb-4a7c-a16e-2d490a2b56b9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3bb8301f69c54c4ebf6f07863f767cd0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:07.696524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7118c76a-fecb-4a7c-a16e-2d490a2b56b9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3bb8301f69c54c4ebf6f07863f767cd0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:07.702108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7118c76a-fecb-4a7c-a16e-2d490a2b56b9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3bb8301f69c54c4ebf6f07863f767cd0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:07.734166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7118c76a-fecb-4a7c-a16e-2d490a2b56b9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3bb8301f69c54c4ebf6f07863f767cd0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:07.738468 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:07.762224 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 882/1748] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:07 2022] DELETE /identity/v3/users/3bb8301f69c54c4ebf6f07863f767cd0 => generated 0 bytes in 568 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:07.767296 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6924b752-8fc3-46bd-a53a-a0759cf684c2 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:07.767816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6924b752-8fc3-46bd-a53a-a0759cf684c2 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:07.767921 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6924b752-8fc3-46bd-a53a-a0759cf684c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:07.834319 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-6924b752-8fc3-46bd-a53a-a0759cf684c2 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:07.839809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6924b752-8fc3-46bd-a53a-a0759cf684c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:08.027437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b967762-ef0a-4f89-9dde-e0d43170d212 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:08.027731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b967762-ef0a-4f89-9dde-e0d43170d212 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:08.028079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b967762-ef0a-4f89-9dde-e0d43170d212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:08.072506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 867/1749] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:08.098769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-9b967762-ef0a-4f89-9dde-e0d43170d212 None None] MFA Rules not processed for user `820f4bb963644499a77fa1f3edd59d08`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:08.100295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-9b967762-ef0a-4f89-9dde-e0d43170d212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:08.261696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 883/1750] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:08 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:09.403188 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6924b752-8fc3-46bd-a53a-a0759cf684c2 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:09.405574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6924b752-8fc3-46bd-a53a-a0759cf684c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:09.598070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:09.707892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6924b752-8fc3-46bd-a53a-a0759cf684c2 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:09.716835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db40d11-3b56-439c-ac3b-daa12c59eb6c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:09.717487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db40d11-3b56-439c-ac3b-daa12c59eb6c service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:09.717818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db40d11-3b56-439c-ac3b-daa12c59eb6c service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:09.718051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db40d11-3b56-439c-ac3b-daa12c59eb6c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:09.752200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db40d11-3b56-439c-ac3b-daa12c59eb6c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:09.756585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db40d11-3b56-439c-ac3b-daa12c59eb6c service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:09.801215 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 868/1751] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:11.362810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e06a08-be0e-40f9-8ef6-a6bf2eabf988 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:11.363381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e06a08-be0e-40f9-8ef6-a6bf2eabf988 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:11.363601 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e06a08-be0e-40f9-8ef6-a6bf2eabf988 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:11.365536 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 884/1752] 10.222.0.26 () {54 vars in 921 bytes} [Wed Sep 28 12:15:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:15:11.373527 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db40d11-3b56-439c-ac3b-daa12c59eb6c service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:11.375056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8db40d11-3b56-439c-ac3b-daa12c59eb6c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:11.395024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38ed774-5bcf-4bb4-ab76-24149ccf4424 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:11.395317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38ed774-5bcf-4bb4-ab76-24149ccf4424 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:11.395598 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38ed774-5bcf-4bb4-ab76-24149ccf4424 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:11.398191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db40d11-3b56-439c-ac3b-daa12c59eb6c service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:11.405218 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:11.405864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:11.406140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:11.406406 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] PATH_INFO: `/v3/projects/5635233bf7394bd199c675fefc0fe603` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:11.407651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5635233bf7394bd199c675fefc0fe603)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:11.410257 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:11.423923 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-a38ed774-5bcf-4bb4-ab76-24149ccf4424 None None] MFA Rules not processed for user `8bf969348e7a4158acf8fb789430b0e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:11.427242 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a38ed774-5bcf-4bb4-ab76-24149ccf4424 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:11.544638 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5635233bf7394bd199c675fefc0fe603', 'request_id': 'req-0e043c8b-f89a-44c0-acfb-02246387dfd8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:11.558875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5635233bf7394bd199c675fefc0fe603', 'request_id': 'req-0e043c8b-f89a-44c0-acfb-02246387dfd8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:11.584643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5635233bf7394bd199c675fefc0fe603', 'request_id': 'req-0e043c8b-f89a-44c0-acfb-02246387dfd8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:11.597659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 885/1753] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Sep 28 12:15:11 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:11.608137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] The token cache is being invalidate because project 5635233bf7394bd199c675fefc0fe603 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:11.608534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:11.616475 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:11.618167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 869/1754] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:11 2022] DELETE /identity/v3/projects/5635233bf7394bd199c675fefc0fe603 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:11.622450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38ed774-5bcf-4bb4-ab76-24149ccf4424 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:11.640036 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38ed774-5bcf-4bb4-ab76-24149ccf4424 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:11.727383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:11.729959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:11.797812 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120ad88b-b22f-4ba3-afe9-1899a2fd2fb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:11.798616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120ad88b-b22f-4ba3-afe9-1899a2fd2fb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:11.799356 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120ad88b-b22f-4ba3-afe9-1899a2fd2fb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:11.799639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120ad88b-b22f-4ba3-afe9-1899a2fd2fb3 admin admin] PATH_INFO: `/v3/users/6554334bf26f4048b8a407be0eb3b73a` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:11.801049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120ad88b-b22f-4ba3-afe9-1899a2fd2fb3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6554334bf26f4048b8a407be0eb3b73a)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:11.802993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120ad88b-b22f-4ba3-afe9-1899a2fd2fb3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:11.887719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:DELETE=6,keystone:SELECT=94 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:11.923712 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-120ad88b-b22f-4ba3-afe9-1899a2fd2fb3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6554334bf26f4048b8a407be0eb3b73a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:11.925345 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-120ad88b-b22f-4ba3-afe9-1899a2fd2fb3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6554334bf26f4048b8a407be0eb3b73a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:11.941774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-120ad88b-b22f-4ba3-afe9-1899a2fd2fb3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6554334bf26f4048b8a407be0eb3b73a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:11.943929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-120ad88b-b22f-4ba3-afe9-1899a2fd2fb3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6554334bf26f4048b8a407be0eb3b73a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:11.975025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-120ad88b-b22f-4ba3-afe9-1899a2fd2fb3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6554334bf26f4048b8a407be0eb3b73a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:12.002330 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e043c8b-f89a-44c0-acfb-02246387dfd8 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:12.010818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8910a79-264a-4cc8-ac28-2efd4a33c876 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:12.011936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8910a79-264a-4cc8-ac28-2efd4a33c876 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:12.012393 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 886/1755] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:11 2022] DELETE /identity/v3/users/6554334bf26f4048b8a407be0eb3b73a => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:12.012508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8910a79-264a-4cc8-ac28-2efd4a33c876 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:12.023706 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8910a79-264a-4cc8-ac28-2efd4a33c876 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:12.028505 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f8910a79-264a-4cc8-ac28-2efd4a33c876 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:12.064946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8910a79-264a-4cc8-ac28-2efd4a33c876 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:12.075873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8910a79-264a-4cc8-ac28-2efd4a33c876 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:12.092711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 870/1756] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:15:12.831479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120ad88b-b22f-4ba3-afe9-1899a2fd2fb3 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:12.833670 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-120ad88b-b22f-4ba3-afe9-1899a2fd2fb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:12.944203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120ad88b-b22f-4ba3-afe9-1899a2fd2fb3 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:12.951501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:12.952383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:12.952711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:12.953012 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf admin admin] PATH_INFO: `/v3/projects/ed9302581e5b4d3b8e043f83cbd3aef9` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:12.954353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf admin admin] RBAC: Authorizing `identity:delete_project(project_id=ed9302581e5b4d3b8e043f83cbd3aef9)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:12.956694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:13.015103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ed9302581e5b4d3b8e043f83cbd3aef9', 'request_id': 'req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:13.021195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ed9302581e5b4d3b8e043f83cbd3aef9', 'request_id': 'req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:13.031620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ed9302581e5b4d3b8e043f83cbd3aef9', 'request_id': 'req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:13.044806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf admin admin] The token cache is being invalidate because project ed9302581e5b4d3b8e043f83cbd3aef9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:13.045035 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:13.045679 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-5f80153b-2d05-4fcc-bff2-9c7b8f2b7aaf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:13.047410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 887/1757] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:12 2022] DELETE /identity/v3/projects/ed9302581e5b4d3b8e043f83cbd3aef9 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:13.055995 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b3cf5e-70de-4988-916c-e9d5f6f9a8cd None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:13.057020 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b3cf5e-70de-4988-916c-e9d5f6f9a8cd None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:13.057485 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b3cf5e-70de-4988-916c-e9d5f6f9a8cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:13.084690 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-49b3cf5e-70de-4988-916c-e9d5f6f9a8cd None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:13.085131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-49b3cf5e-70de-4988-916c-e9d5f6f9a8cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:13.134037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:13.177071 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 871/1758] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:13.186243 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d37af2-6aa1-4bc8-9d38-6646817134a2 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:13.186626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d37af2-6aa1-4bc8-9d38-6646817134a2 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:13.187215 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d37af2-6aa1-4bc8-9d38-6646817134a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:13.211112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-75d37af2-6aa1-4bc8-9d38-6646817134a2 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:13.212578 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-75d37af2-6aa1-4bc8-9d38-6646817134a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:13.221022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 888/1759] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:13.225370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b3cf5e-70de-4988-916c-e9d5f6f9a8cd None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:13.236009 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b3cf5e-70de-4988-916c-e9d5f6f9a8cd None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:13.241061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcd45d8-d967-4a36-81be-cc52a6839877 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:13.241749 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcd45d8-d967-4a36-81be-cc52a6839877 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:13.242066 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcd45d8-d967-4a36-81be-cc52a6839877 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:13.242343 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcd45d8-d967-4a36-81be-cc52a6839877 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:13.242997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcd45d8-d967-4a36-81be-cc52a6839877 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:15:13.243301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcd45d8-d967-4a36-81be-cc52a6839877 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:13.245082 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcd45d8-d967-4a36-81be-cc52a6839877 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:13.257576 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 872/1760] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:15:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:15:13.262027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d37af2-6aa1-4bc8-9d38-6646817134a2 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:13.272197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d37af2-6aa1-4bc8-9d38-6646817134a2 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:13.277411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6c6f6b-832a-4c2e-a5e2-9f9d3a133d59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:13.278084 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6c6f6b-832a-4c2e-a5e2-9f9d3a133d59 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:13.278431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6c6f6b-832a-4c2e-a5e2-9f9d3a133d59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:13.278774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6c6f6b-832a-4c2e-a5e2-9f9d3a133d59 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:13.280060 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6c6f6b-832a-4c2e-a5e2-9f9d3a133d59 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:13.282075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6c6f6b-832a-4c2e-a5e2-9f9d3a133d59 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:13.313568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 889/1761] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:13 2022] POST /identity/v3/projects => generated 407 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:13.315632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcd45d8-d967-4a36-81be-cc52a6839877 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:13.326736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcd45d8-d967-4a36-81be-cc52a6839877 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:13.387212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deba7157-3f8a-4e70-b9ff-5f5ea51414de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:13.387822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deba7157-3f8a-4e70-b9ff-5f5ea51414de admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:13.388102 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deba7157-3f8a-4e70-b9ff-5f5ea51414de admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:13.388410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deba7157-3f8a-4e70-b9ff-5f5ea51414de admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:13.389083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deba7157-3f8a-4e70-b9ff-5f5ea51414de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:13.391242 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deba7157-3f8a-4e70-b9ff-5f5ea51414de admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:13.423634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 873/1762] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:13 2022] POST /identity/v3/users => generated 304 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:13.425018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6c6f6b-832a-4c2e-a5e2-9f9d3a133d59 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:13.436357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6c6f6b-832a-4c2e-a5e2-9f9d3a133d59 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:13.441180 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2d9bf9-fdc5-4119-9642-0a56698ff7a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:13.441789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2d9bf9-fdc5-4119-9642-0a56698ff7a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:13.442066 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2d9bf9-fdc5-4119-9642-0a56698ff7a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:13.442455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2d9bf9-fdc5-4119-9642-0a56698ff7a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:13.443116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2d9bf9-fdc5-4119-9642-0a56698ff7a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:13.444751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2d9bf9-fdc5-4119-9642-0a56698ff7a1 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:13.450271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:13.456906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 890/1763] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:13 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:13.460468 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deba7157-3f8a-4e70-b9ff-5f5ea51414de admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:13.471483 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deba7157-3f8a-4e70-b9ff-5f5ea51414de admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:13.476136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bdf077-3b8a-48e0-9f99-e8430ed32b4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:13.476778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bdf077-3b8a-48e0-9f99-e8430ed32b4b admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:13.477054 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bdf077-3b8a-48e0-9f99-e8430ed32b4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:13.477323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bdf077-3b8a-48e0-9f99-e8430ed32b4b admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:13.477997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bdf077-3b8a-48e0-9f99-e8430ed32b4b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:13.479594 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bdf077-3b8a-48e0-9f99-e8430ed32b4b admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:13.490578 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 874/1764] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:13 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:13.495031 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2d9bf9-fdc5-4119-9642-0a56698ff7a1 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:13.508797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2d9bf9-fdc5-4119-9642-0a56698ff7a1 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:13.514861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adacc862-67e0-4d62-925d-8ab8c26c83e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:13.515637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adacc862-67e0-4d62-925d-8ab8c26c83e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:13.516030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adacc862-67e0-4d62-925d-8ab8c26c83e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:13.516437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adacc862-67e0-4d62-925d-8ab8c26c83e5 admin admin] PATH_INFO: `/v3/projects/b287e1f53b0749c5904b1e7ef25b3b46/users/772a9a9fda4c446fa3d08eaae1ab9c82/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:13.534199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adacc862-67e0-4d62-925d-8ab8c26c83e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b287e1f53b0749c5904b1e7ef25b3b46, user_id=772a9a9fda4c446fa3d08eaae1ab9c82, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:13.537591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adacc862-67e0-4d62-925d-8ab8c26c83e5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:13.552944 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 891/1765] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:13 2022] PUT /identity/v3/projects/b287e1f53b0749c5904b1e7ef25b3b46/users/772a9a9fda4c446fa3d08eaae1ab9c82/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:13.563783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf2c3dd-4935-4c5d-bcf3-087ecd2241a9 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:13.564097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf2c3dd-4935-4c5d-bcf3-087ecd2241a9 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:13.564361 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf2c3dd-4935-4c5d-bcf3-087ecd2241a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:13.596212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-1bf2c3dd-4935-4c5d-bcf3-087ecd2241a9 None None] MFA Rules not processed for user `772a9a9fda4c446fa3d08eaae1ab9c82`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:13.597486 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1bf2c3dd-4935-4c5d-bcf3-087ecd2241a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:13.677634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 875/1766] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:13 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:13.682001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adacc862-67e0-4d62-925d-8ab8c26c83e5 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:13.752998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adacc862-67e0-4d62-925d-8ab8c26c83e5 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:13.758343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f338ec1f-5098-4d3e-a1cc-9c672816df5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:13.759049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f338ec1f-5098-4d3e-a1cc-9c672816df5c admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:13.759457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f338ec1f-5098-4d3e-a1cc-9c672816df5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:13.759795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f338ec1f-5098-4d3e-a1cc-9c672816df5c admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:13.760535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f338ec1f-5098-4d3e-a1cc-9c672816df5c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:13.762513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f338ec1f-5098-4d3e-a1cc-9c672816df5c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:13.792507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 892/1767] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:13 2022] POST /identity/v3/projects => generated 407 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:15:13.793739 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf2c3dd-4935-4c5d-bcf3-087ecd2241a9 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:13.805137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf2c3dd-4935-4c5d-bcf3-087ecd2241a9 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:13.841290 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:13.901832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422f8d38-d003-4b51-acb4-8659fc783f20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:13.902771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422f8d38-d003-4b51-acb4-8659fc783f20 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:13.903262 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422f8d38-d003-4b51-acb4-8659fc783f20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:13.903736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422f8d38-d003-4b51-acb4-8659fc783f20 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:13.905229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422f8d38-d003-4b51-acb4-8659fc783f20 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:13.907680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422f8d38-d003-4b51-acb4-8659fc783f20 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:13.942953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 876/1768] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:13 2022] POST /identity/v3/users => generated 310 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:13.946602 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f338ec1f-5098-4d3e-a1cc-9c672816df5c admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:13.971674 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f338ec1f-5098-4d3e-a1cc-9c672816df5c admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:13.989311 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236a97d6-1119-44e4-a6d6-f3ca7e4633b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:13.989978 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236a97d6-1119-44e4-a6d6-f3ca7e4633b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:13.990259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236a97d6-1119-44e4-a6d6-f3ca7e4633b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:13.990532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236a97d6-1119-44e4-a6d6-f3ca7e4633b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:13.991263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236a97d6-1119-44e4-a6d6-f3ca7e4633b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:14.000245 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236a97d6-1119-44e4-a6d6-f3ca7e4633b2 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:14.023599 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 893/1769] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:13 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:14.028007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422f8d38-d003-4b51-acb4-8659fc783f20 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:14.042964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422f8d38-d003-4b51-acb4-8659fc783f20 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:14.053641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733de8fb-f812-4c30-ba7e-b8546270b7bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:14.054183 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733de8fb-f812-4c30-ba7e-b8546270b7bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:14.054464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733de8fb-f812-4c30-ba7e-b8546270b7bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:14.054742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733de8fb-f812-4c30-ba7e-b8546270b7bb admin admin] PATH_INFO: `/v3/projects/78ab99f10fb64157818f4d5444f2a7bf/users/3716a572b8a84368bdbe0163900048f2/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:14.079638 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733de8fb-f812-4c30-ba7e-b8546270b7bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=78ab99f10fb64157818f4d5444f2a7bf, user_id=3716a572b8a84368bdbe0163900048f2, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:14.082822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733de8fb-f812-4c30-ba7e-b8546270b7bb admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:14.101021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 877/1770] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:14 2022] PUT /identity/v3/projects/78ab99f10fb64157818f4d5444f2a7bf/users/3716a572b8a84368bdbe0163900048f2/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:15:14.104968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236a97d6-1119-44e4-a6d6-f3ca7e4633b2 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:14.116986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236a97d6-1119-44e4-a6d6-f3ca7e4633b2 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:14.180645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c947b089-1266-4258-9c23-90fad0ec7aed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:14.181262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c947b089-1266-4258-9c23-90fad0ec7aed admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:14.181540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c947b089-1266-4258-9c23-90fad0ec7aed admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:14.181812 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c947b089-1266-4258-9c23-90fad0ec7aed admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:14.182462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c947b089-1266-4258-9c23-90fad0ec7aed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:14.184033 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c947b089-1266-4258-9c23-90fad0ec7aed admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:14.198613 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 894/1771] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:14 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:15:14.200256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733de8fb-f812-4c30-ba7e-b8546270b7bb admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:14.217528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733de8fb-f812-4c30-ba7e-b8546270b7bb admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:14.223090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e34746a-7b18-4b87-9499-37aa5817960a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:14.223877 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e34746a-7b18-4b87-9499-37aa5817960a admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:14.224226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e34746a-7b18-4b87-9499-37aa5817960a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:14.224701 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e34746a-7b18-4b87-9499-37aa5817960a admin admin] PATH_INFO: `/v3/projects/78ab99f10fb64157818f4d5444f2a7bf/users/3716a572b8a84368bdbe0163900048f2/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:14.226634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e34746a-7b18-4b87-9499-37aa5817960a admin admin] RBAC: Authorizing `identity:create_grant(project_id=78ab99f10fb64157818f4d5444f2a7bf, user_id=3716a572b8a84368bdbe0163900048f2, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:14.229366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e34746a-7b18-4b87-9499-37aa5817960a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:14.239845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 878/1772] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:14 2022] PUT /identity/v3/projects/78ab99f10fb64157818f4d5444f2a7bf/users/3716a572b8a84368bdbe0163900048f2/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:14.251887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864d5af5-069b-4ed4-8e61-3f25961c4f26 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:14.252178 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864d5af5-069b-4ed4-8e61-3f25961c4f26 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:14.253300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864d5af5-069b-4ed4-8e61-3f25961c4f26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:14.283194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-864d5af5-069b-4ed4-8e61-3f25961c4f26 None None] MFA Rules not processed for user `3716a572b8a84368bdbe0163900048f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:14.287852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-864d5af5-069b-4ed4-8e61-3f25961c4f26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:14.334253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:14.379954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 895/1773] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:14 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:14.391336 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e34746a-7b18-4b87-9499-37aa5817960a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:14.393419 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3e34746a-7b18-4b87-9499-37aa5817960a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:14.512876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e34746a-7b18-4b87-9499-37aa5817960a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:14.517699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5451bc-2925-4563-8f28-0aa2aa420713 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:14.518314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5451bc-2925-4563-8f28-0aa2aa420713 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:14.518596 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5451bc-2925-4563-8f28-0aa2aa420713 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:14.518872 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5451bc-2925-4563-8f28-0aa2aa420713 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:14.535644 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5451bc-2925-4563-8f28-0aa2aa420713 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:14.537280 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5451bc-2925-4563-8f28-0aa2aa420713 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:14.557956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 879/1774] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:14 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:14.825133 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864d5af5-069b-4ed4-8e61-3f25961c4f26 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:14.842390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864d5af5-069b-4ed4-8e61-3f25961c4f26 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:14.853950 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d25450-e1fb-4465-9b0c-28a2f210afa4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:14.854376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d25450-e1fb-4465-9b0c-28a2f210afa4 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:14.858721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d25450-e1fb-4465-9b0c-28a2f210afa4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:14.859212 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d25450-e1fb-4465-9b0c-28a2f210afa4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:14.989884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d25450-e1fb-4465-9b0c-28a2f210afa4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:14.991458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d25450-e1fb-4465-9b0c-28a2f210afa4 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:15.002050 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 896/1775] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:14 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:16.137836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5451bc-2925-4563-8f28-0aa2aa420713 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:16.139727 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8e5451bc-2925-4563-8f28-0aa2aa420713 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:16.287744 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5451bc-2925-4563-8f28-0aa2aa420713 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:16.294293 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d40a5c9-19ff-4d70-bc14-ae28f657487f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:16.296132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d40a5c9-19ff-4d70-bc14-ae28f657487f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:16.296907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d40a5c9-19ff-4d70-bc14-ae28f657487f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:16.297389 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d40a5c9-19ff-4d70-bc14-ae28f657487f admin admin] PATH_INFO: `/v3/users/d90c463f98244312b78e998f4df89188` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:16.298875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d40a5c9-19ff-4d70-bc14-ae28f657487f admin admin] RBAC: Authorizing `identity:delete_user(user_id=d90c463f98244312b78e998f4df89188)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:16.301076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d40a5c9-19ff-4d70-bc14-ae28f657487f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:16.355573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:16.415377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-9d40a5c9-19ff-4d70-bc14-ae28f657487f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd90c463f98244312b78e998f4df89188'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:16.416681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-9d40a5c9-19ff-4d70-bc14-ae28f657487f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd90c463f98244312b78e998f4df89188'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:16.417694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-9d40a5c9-19ff-4d70-bc14-ae28f657487f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd90c463f98244312b78e998f4df89188'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:16.437446 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-9d40a5c9-19ff-4d70-bc14-ae28f657487f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd90c463f98244312b78e998f4df89188'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:16.500011 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-9d40a5c9-19ff-4d70-bc14-ae28f657487f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd90c463f98244312b78e998f4df89188'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:16.541600 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 880/1776] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:16 2022] DELETE /identity/v3/users/d90c463f98244312b78e998f4df89188 => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:16.546500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c3f4a1-a537-4ca3-bbd6-73ff39295787 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:16.547369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c3f4a1-a537-4ca3-bbd6-73ff39295787 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:16.547966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c3f4a1-a537-4ca3-bbd6-73ff39295787 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:16.599246 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-92c3f4a1-a537-4ca3-bbd6-73ff39295787 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:16.601254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-92c3f4a1-a537-4ca3-bbd6-73ff39295787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:16.883816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 897/1777] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 339 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:16.968582 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d40a5c9-19ff-4d70-bc14-ae28f657487f admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:16.970366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9d40a5c9-19ff-4d70-bc14-ae28f657487f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:17.226905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d40a5c9-19ff-4d70-bc14-ae28f657487f admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:17.282145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca6dbac-8841-43fa-a709-e12337c7778c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:17.283390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca6dbac-8841-43fa-a709-e12337c7778c service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:17.283845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca6dbac-8841-43fa-a709-e12337c7778c service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:17.284276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca6dbac-8841-43fa-a709-e12337c7778c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:17.302741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca6dbac-8841-43fa-a709-e12337c7778c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:17.305294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca6dbac-8841-43fa-a709-e12337c7778c service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:17.317364 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 881/1778] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:18.797952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c3f4a1-a537-4ca3-bbd6-73ff39295787 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:18.800624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-92c3f4a1-a537-4ca3-bbd6-73ff39295787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:18.829485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c3f4a1-a537-4ca3-bbd6-73ff39295787 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:18.845059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:18.845989 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:18.846434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:18.847400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] PATH_INFO: `/v3/projects/66c178066d9e4b86835ed533fcd680cb` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:18.849251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=66c178066d9e4b86835ed533fcd680cb)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:18.854618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:18.931284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '66c178066d9e4b86835ed533fcd680cb', 'request_id': 'req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:18.941819 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '66c178066d9e4b86835ed533fcd680cb', 'request_id': 'req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:18.954239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '66c178066d9e4b86835ed533fcd680cb', 'request_id': 'req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:18.956135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:18.980933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daade87a-3f84-458e-8ac5-74ff0d8b51e1 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:18.981546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daade87a-3f84-458e-8ac5-74ff0d8b51e1 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:18.981646 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daade87a-3f84-458e-8ac5-74ff0d8b51e1 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:18.984153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 882/1779] 10.222.0.26 () {54 vars in 921 bytes} [Wed Sep 28 12:15:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:15:18.986067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] The token cache is being invalidate because project 66c178066d9e4b86835ed533fcd680cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:18.986370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:18.987261 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:18.989207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03411be-0073-4c7d-8f3d-a57af48118d7 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:18.989838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 898/1780] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:18 2022] DELETE /identity/v3/projects/66c178066d9e4b86835ed533fcd680cb => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:18.993451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03411be-0073-4c7d-8f3d-a57af48118d7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:18.993891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03411be-0073-4c7d-8f3d-a57af48118d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:18.994818 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:18.996389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:19.020049 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-d03411be-0073-4c7d-8f3d-a57af48118d7 None None] MFA Rules not processed for user `8bf969348e7a4158acf8fb789430b0e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:19.021635 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d03411be-0073-4c7d-8f3d-a57af48118d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:19.078345 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25055ca-f2d9-4d4b-b2eb-02d0ba783ff3 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:19.085375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51defabe-7460-438c-985a-1c73c2c9d1a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:19.086178 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51defabe-7460-438c-985a-1c73c2c9d1a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:19.086465 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51defabe-7460-438c-985a-1c73c2c9d1a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:19.086765 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51defabe-7460-438c-985a-1c73c2c9d1a2 admin admin] PATH_INFO: `/v3/users/05a44c3e1dce43c48b91a53b6f3e15d2` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:19.087956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51defabe-7460-438c-985a-1c73c2c9d1a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=05a44c3e1dce43c48b91a53b6f3e15d2)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:19.090022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51defabe-7460-438c-985a-1c73c2c9d1a2 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:19.127569 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 883/1781] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Sep 28 12:15:18 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:19.172413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-51defabe-7460-438c-985a-1c73c2c9d1a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05a44c3e1dce43c48b91a53b6f3e15d2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:19.173377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-51defabe-7460-438c-985a-1c73c2c9d1a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05a44c3e1dce43c48b91a53b6f3e15d2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:19.197501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-51defabe-7460-438c-985a-1c73c2c9d1a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05a44c3e1dce43c48b91a53b6f3e15d2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:19.199140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-51defabe-7460-438c-985a-1c73c2c9d1a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05a44c3e1dce43c48b91a53b6f3e15d2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:19.203902 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03411be-0073-4c7d-8f3d-a57af48118d7 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:19.205510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d03411be-0073-4c7d-8f3d-a57af48118d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:19.212409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-51defabe-7460-438c-985a-1c73c2c9d1a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05a44c3e1dce43c48b91a53b6f3e15d2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:19.220207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03411be-0073-4c7d-8f3d-a57af48118d7 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:19.226478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79e6307-91f0-4dca-a88c-6eb05dae0067 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:19.227110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79e6307-91f0-4dca-a88c-6eb05dae0067 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:19.227388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79e6307-91f0-4dca-a88c-6eb05dae0067 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:19.227661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79e6307-91f0-4dca-a88c-6eb05dae0067 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:19.231085 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 899/1782] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:18 2022] DELETE /identity/v3/users/05a44c3e1dce43c48b91a53b6f3e15d2 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:19.231213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b79e6307-91f0-4dca-a88c-6eb05dae0067 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:19.244937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79e6307-91f0-4dca-a88c-6eb05dae0067 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:19.247561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79e6307-91f0-4dca-a88c-6eb05dae0067 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:19.260257 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 884/1783] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:19.515463 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51defabe-7460-438c-985a-1c73c2c9d1a2 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:19.516918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-51defabe-7460-438c-985a-1c73c2c9d1a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:19.587081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51defabe-7460-438c-985a-1c73c2c9d1a2 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:19.592492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6943351-9448-4d6a-8263-5a58d6aea458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:19.593128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6943351-9448-4d6a-8263-5a58d6aea458 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:19.593407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6943351-9448-4d6a-8263-5a58d6aea458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:19.593923 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6943351-9448-4d6a-8263-5a58d6aea458 admin admin] PATH_INFO: `/v3/projects/8d330d3d8c024696a812f028d3509bf9` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:19.595082 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6943351-9448-4d6a-8263-5a58d6aea458 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d330d3d8c024696a812f028d3509bf9)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:19.597005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6943351-9448-4d6a-8263-5a58d6aea458 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:19.613096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:19.643507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-e6943351-9448-4d6a-8263-5a58d6aea458 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d330d3d8c024696a812f028d3509bf9', 'request_id': 'req-e6943351-9448-4d6a-8263-5a58d6aea458'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:19.648951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-e6943351-9448-4d6a-8263-5a58d6aea458 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d330d3d8c024696a812f028d3509bf9', 'request_id': 'req-e6943351-9448-4d6a-8263-5a58d6aea458'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:19.655828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-e6943351-9448-4d6a-8263-5a58d6aea458 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d330d3d8c024696a812f028d3509bf9', 'request_id': 'req-e6943351-9448-4d6a-8263-5a58d6aea458'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:19.666913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-e6943351-9448-4d6a-8263-5a58d6aea458 admin admin] The token cache is being invalidate because project 8d330d3d8c024696a812f028d3509bf9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:19.667200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-e6943351-9448-4d6a-8263-5a58d6aea458 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:19.667826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-e6943351-9448-4d6a-8263-5a58d6aea458 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:19.669481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 900/1784] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:19 2022] DELETE /identity/v3/projects/8d330d3d8c024696a812f028d3509bf9 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:19.674800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0744474-7fe7-47ba-8a2a-53bd1b7d096f None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:19.675089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0744474-7fe7-47ba-8a2a-53bd1b7d096f None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:19.675369 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0744474-7fe7-47ba-8a2a-53bd1b7d096f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:19.694930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-b0744474-7fe7-47ba-8a2a-53bd1b7d096f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:19.696122 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b0744474-7fe7-47ba-8a2a-53bd1b7d096f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:19.715724 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:19.755543 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 885/1785] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:19.763270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5d987-2ee2-40a9-a5ae-6c28b9d962ba None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:19.763579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5d987-2ee2-40a9-a5ae-6c28b9d962ba None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:19.763854 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5d987-2ee2-40a9-a5ae-6c28b9d962ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:19.782787 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-11e5d987-2ee2-40a9-a5ae-6c28b9d962ba None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:19.783975 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-11e5d987-2ee2-40a9-a5ae-6c28b9d962ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:19.791627 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 901/1786] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:19.796100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0744474-7fe7-47ba-8a2a-53bd1b7d096f None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:19.805608 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0744474-7fe7-47ba-8a2a-53bd1b7d096f None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:19.810184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51726952-ed67-4ad8-8698-f429bdafba04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:19.811363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51726952-ed67-4ad8-8698-f429bdafba04 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:19.811695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51726952-ed67-4ad8-8698-f429bdafba04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:19.811971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51726952-ed67-4ad8-8698-f429bdafba04 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:19.812668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51726952-ed67-4ad8-8698-f429bdafba04 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:15:19.812974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51726952-ed67-4ad8-8698-f429bdafba04 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:19.814588 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51726952-ed67-4ad8-8698-f429bdafba04 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:19.826190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 886/1787] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:15:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:15:19.830384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e5d987-2ee2-40a9-a5ae-6c28b9d962ba None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:19.839402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e5d987-2ee2-40a9-a5ae-6c28b9d962ba None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:19.844284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e9105c-69a8-44e5-acb5-ffbf5e41fc85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:19.844966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e9105c-69a8-44e5-acb5-ffbf5e41fc85 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:19.845244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e9105c-69a8-44e5-acb5-ffbf5e41fc85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:19.845513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e9105c-69a8-44e5-acb5-ffbf5e41fc85 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:19.846133 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e9105c-69a8-44e5-acb5-ffbf5e41fc85 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:19.848007 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e9105c-69a8-44e5-acb5-ffbf5e41fc85 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:19.875256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 902/1788] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:19 2022] POST /identity/v3/projects => generated 369 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:19.879236 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51726952-ed67-4ad8-8698-f429bdafba04 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:19.888952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51726952-ed67-4ad8-8698-f429bdafba04 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:19.943615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0223c5-fbba-4794-b3e5-2445fba05cc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:19.944139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0223c5-fbba-4794-b3e5-2445fba05cc0 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:19.944455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0223c5-fbba-4794-b3e5-2445fba05cc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:19.944736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0223c5-fbba-4794-b3e5-2445fba05cc0 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:19.945364 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0223c5-fbba-4794-b3e5-2445fba05cc0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:19.947312 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0223c5-fbba-4794-b3e5-2445fba05cc0 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:19.971111 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 887/1789] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:19 2022] POST /identity/v3/users => generated 285 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:19.978526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e9105c-69a8-44e5-acb5-ffbf5e41fc85 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:19.993011 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e9105c-69a8-44e5-acb5-ffbf5e41fc85 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:19.999358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a2486f-0277-41e3-bdd1-158b753a5ae0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:20.000127 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a2486f-0277-41e3-bdd1-158b753a5ae0 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:20.000559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a2486f-0277-41e3-bdd1-158b753a5ae0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:20.001015 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a2486f-0277-41e3-bdd1-158b753a5ae0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:20.001784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a2486f-0277-41e3-bdd1-158b753a5ae0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:20.003496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a2486f-0277-41e3-bdd1-158b753a5ae0 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:20.014890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 903/1790] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:19 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:20.019002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0223c5-fbba-4794-b3e5-2445fba05cc0 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:20.031208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0223c5-fbba-4794-b3e5-2445fba05cc0 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:20.036031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0307ea-eedc-4c0e-bc93-24bc4d4eb963 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:20.036667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0307ea-eedc-4c0e-bc93-24bc4d4eb963 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:20.036951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0307ea-eedc-4c0e-bc93-24bc4d4eb963 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:20.037229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0307ea-eedc-4c0e-bc93-24bc4d4eb963 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:20.037873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0307ea-eedc-4c0e-bc93-24bc4d4eb963 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:20.039491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0307ea-eedc-4c0e-bc93-24bc4d4eb963 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:20.050645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 888/1791] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:20.054899 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a2486f-0277-41e3-bdd1-158b753a5ae0 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:20.065986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a2486f-0277-41e3-bdd1-158b753a5ae0 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:20.070846 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea23f16-0108-4802-a177-85661379e3f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:20.071622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea23f16-0108-4802-a177-85661379e3f6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:20.071967 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea23f16-0108-4802-a177-85661379e3f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:20.072493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea23f16-0108-4802-a177-85661379e3f6 admin admin] PATH_INFO: `/v3/projects/4ac8da1514e14849948adf4034874003/users/6bf836f02e6749b09a3a023a675613f1/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:20.089768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea23f16-0108-4802-a177-85661379e3f6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ac8da1514e14849948adf4034874003, user_id=6bf836f02e6749b09a3a023a675613f1, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:20.093035 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea23f16-0108-4802-a177-85661379e3f6 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:20.107694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 904/1792] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:20 2022] PUT /identity/v3/projects/4ac8da1514e14849948adf4034874003/users/6bf836f02e6749b09a3a023a675613f1/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:15:20.119324 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed4c9de-c7cf-4805-848c-7cd47a372ab3 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:20.119666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed4c9de-c7cf-4805-848c-7cd47a372ab3 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:20.119946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed4c9de-c7cf-4805-848c-7cd47a372ab3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:20.152081 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-8ed4c9de-c7cf-4805-848c-7cd47a372ab3 None None] MFA Rules not processed for user `6bf836f02e6749b09a3a023a675613f1`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:20.153391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8ed4c9de-c7cf-4805-848c-7cd47a372ab3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:20.232879 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 889/1793] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:20 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:20.233331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea23f16-0108-4802-a177-85661379e3f6 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:20.317009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea23f16-0108-4802-a177-85661379e3f6 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:20.339895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0d7ad8-1234-4831-85bf-7f1e6b7f10ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:20.340931 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0d7ad8-1234-4831-85bf-7f1e6b7f10ef admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:20.341319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0d7ad8-1234-4831-85bf-7f1e6b7f10ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:20.341850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0d7ad8-1234-4831-85bf-7f1e6b7f10ef admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:20.342752 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0d7ad8-1234-4831-85bf-7f1e6b7f10ef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:20.345084 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0d7ad8-1234-4831-85bf-7f1e6b7f10ef admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:20.405002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 905/1794] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:20 2022] POST /identity/v3/projects => generated 367 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:15:20.409142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed4c9de-c7cf-4805-848c-7cd47a372ab3 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:20.424156 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed4c9de-c7cf-4805-848c-7cd47a372ab3 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:20.475500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:20.524762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b290a0-c843-4211-9705-055479e42b21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:20.525570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b290a0-c843-4211-9705-055479e42b21 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:20.525953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b290a0-c843-4211-9705-055479e42b21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:20.526331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b290a0-c843-4211-9705-055479e42b21 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:20.527136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b290a0-c843-4211-9705-055479e42b21 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:20.529348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b290a0-c843-4211-9705-055479e42b21 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:20.570677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 890/1795] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:20 2022] POST /identity/v3/users => generated 290 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:15:20.590260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0d7ad8-1234-4831-85bf-7f1e6b7f10ef admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:20.602309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0d7ad8-1234-4831-85bf-7f1e6b7f10ef admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:20.608355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffebffb-8a5a-4a6c-944a-bb5dd8768d1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:20.608995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffebffb-8a5a-4a6c-944a-bb5dd8768d1d admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:20.609284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffebffb-8a5a-4a6c-944a-bb5dd8768d1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:20.609561 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffebffb-8a5a-4a6c-944a-bb5dd8768d1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:20.610987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffebffb-8a5a-4a6c-944a-bb5dd8768d1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:20.612243 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffebffb-8a5a-4a6c-944a-bb5dd8768d1d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:20.625130 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 906/1796] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:20 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:20.629172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b290a0-c843-4211-9705-055479e42b21 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:20.650399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b290a0-c843-4211-9705-055479e42b21 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:20.667942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa0b3fa-59b3-45d9-8785-bb3fe331da88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:20.668654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa0b3fa-59b3-45d9-8785-bb3fe331da88 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:20.668938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa0b3fa-59b3-45d9-8785-bb3fe331da88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:20.669218 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa0b3fa-59b3-45d9-8785-bb3fe331da88 admin admin] PATH_INFO: `/v3/projects/4d0b47a8cdb34e0daf711a2f1958c121/users/be781cd70bc9415f813659b7b3783f8c/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:20.690057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa0b3fa-59b3-45d9-8785-bb3fe331da88 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d0b47a8cdb34e0daf711a2f1958c121, user_id=be781cd70bc9415f813659b7b3783f8c, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:20.692772 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa0b3fa-59b3-45d9-8785-bb3fe331da88 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:20.708649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 891/1797] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:20 2022] PUT /identity/v3/projects/4d0b47a8cdb34e0daf711a2f1958c121/users/be781cd70bc9415f813659b7b3783f8c/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:20.717395 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffebffb-8a5a-4a6c-944a-bb5dd8768d1d admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:20.729599 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffebffb-8a5a-4a6c-944a-bb5dd8768d1d admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:20.755115 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:20.810129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be338c7f-010c-4e71-b112-96e189b4ddeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:20.811282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be338c7f-010c-4e71-b112-96e189b4ddeb admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:20.811579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be338c7f-010c-4e71-b112-96e189b4ddeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:20.811851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be338c7f-010c-4e71-b112-96e189b4ddeb admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:20.812546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be338c7f-010c-4e71-b112-96e189b4ddeb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:20.814117 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be338c7f-010c-4e71-b112-96e189b4ddeb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:20.824150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 907/1798] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:20 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:20.830025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa0b3fa-59b3-45d9-8785-bb3fe331da88 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:20.841876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa0b3fa-59b3-45d9-8785-bb3fe331da88 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:20.848786 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba724479-4797-406d-82a1-40671b3b6b18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:20.849440 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba724479-4797-406d-82a1-40671b3b6b18 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:20.849718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba724479-4797-406d-82a1-40671b3b6b18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:20.849992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba724479-4797-406d-82a1-40671b3b6b18 admin admin] PATH_INFO: `/v3/projects/4d0b47a8cdb34e0daf711a2f1958c121/users/be781cd70bc9415f813659b7b3783f8c/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:20.851930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba724479-4797-406d-82a1-40671b3b6b18 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d0b47a8cdb34e0daf711a2f1958c121, user_id=be781cd70bc9415f813659b7b3783f8c, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:20.854694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba724479-4797-406d-82a1-40671b3b6b18 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:20.866487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 892/1799] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:20 2022] PUT /identity/v3/projects/4d0b47a8cdb34e0daf711a2f1958c121/users/be781cd70bc9415f813659b7b3783f8c/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:20.878281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226f88bc-0d83-4ae8-9910-ccf95c1199bb None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:20.878841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226f88bc-0d83-4ae8-9910-ccf95c1199bb None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:20.878841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226f88bc-0d83-4ae8-9910-ccf95c1199bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:20.916250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-226f88bc-0d83-4ae8-9910-ccf95c1199bb None None] MFA Rules not processed for user `be781cd70bc9415f813659b7b3783f8c`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:20.917861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-226f88bc-0d83-4ae8-9910-ccf95c1199bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:20.994885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 908/1800] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:20 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:21.006274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba724479-4797-406d-82a1-40671b3b6b18 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:21.008380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-ba724479-4797-406d-82a1-40671b3b6b18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:21.076959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba724479-4797-406d-82a1-40671b3b6b18 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:21.076959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96338cb-5639-47c0-85a0-1de0533b2575 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:21.076959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96338cb-5639-47c0-85a0-1de0533b2575 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:21.076959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96338cb-5639-47c0-85a0-1de0533b2575 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:21.077914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96338cb-5639-47c0-85a0-1de0533b2575 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:21.090085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96338cb-5639-47c0-85a0-1de0533b2575 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:21.092550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96338cb-5639-47c0-85a0-1de0533b2575 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:21.104125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 893/1801] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:21 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:22.598530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226f88bc-0d83-4ae8-9910-ccf95c1199bb None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:22.668281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226f88bc-0d83-4ae8-9910-ccf95c1199bb None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:22.673557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1788292-f8a7-4108-89fa-ccc22b63782e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:22.674600 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1788292-f8a7-4108-89fa-ccc22b63782e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:22.674961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1788292-f8a7-4108-89fa-ccc22b63782e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:22.675332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1788292-f8a7-4108-89fa-ccc22b63782e admin admin] PATH_INFO: `/v3/users/6bf836f02e6749b09a3a023a675613f1` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:22.676745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1788292-f8a7-4108-89fa-ccc22b63782e admin admin] RBAC: Authorizing `identity:delete_user(user_id=6bf836f02e6749b09a3a023a675613f1)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:22.678786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1788292-f8a7-4108-89fa-ccc22b63782e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:22.738932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:22.749941 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c1788292-f8a7-4108-89fa-ccc22b63782e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6bf836f02e6749b09a3a023a675613f1'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:22.751234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c1788292-f8a7-4108-89fa-ccc22b63782e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6bf836f02e6749b09a3a023a675613f1'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:22.759918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c1788292-f8a7-4108-89fa-ccc22b63782e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6bf836f02e6749b09a3a023a675613f1'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:22.760795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c1788292-f8a7-4108-89fa-ccc22b63782e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6bf836f02e6749b09a3a023a675613f1'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:22.774948 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c1788292-f8a7-4108-89fa-ccc22b63782e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6bf836f02e6749b09a3a023a675613f1'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:22.792189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 909/1802] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:22 2022] DELETE /identity/v3/users/6bf836f02e6749b09a3a023a675613f1 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:22.797206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53460a63-1fb0-4f79-97e5-42496c734f89 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:22.797761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53460a63-1fb0-4f79-97e5-42496c734f89 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:22.797761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53460a63-1fb0-4f79-97e5-42496c734f89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:22.818820 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-53460a63-1fb0-4f79-97e5-42496c734f89 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:22.820171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-53460a63-1fb0-4f79-97e5-42496c734f89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:22.909138 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 894/1803] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:22.918968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1788292-f8a7-4108-89fa-ccc22b63782e admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:22.920950 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c1788292-f8a7-4108-89fa-ccc22b63782e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:22.987470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1788292-f8a7-4108-89fa-ccc22b63782e admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:22.991010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192ccfde-0b76-4c19-8032-6c05da3c99a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:22.991606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192ccfde-0b76-4c19-8032-6c05da3c99a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:22.991882 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192ccfde-0b76-4c19-8032-6c05da3c99a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:22.992155 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192ccfde-0b76-4c19-8032-6c05da3c99a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:23.001439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192ccfde-0b76-4c19-8032-6c05da3c99a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:23.002850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192ccfde-0b76-4c19-8032-6c05da3c99a8 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:23.011956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 910/1804] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:23.288571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53460a63-1fb0-4f79-97e5-42496c734f89 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:23.289986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-53460a63-1fb0-4f79-97e5-42496c734f89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:23.306216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53460a63-1fb0-4f79-97e5-42496c734f89 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:23.311709 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:23.312376 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:23.312653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:23.313024 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] PATH_INFO: `/v3/projects/4ac8da1514e14849948adf4034874003` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:23.314326 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4ac8da1514e14849948adf4034874003)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:23.316223 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:23.328415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:23.365227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4ac8da1514e14849948adf4034874003', 'request_id': 'req-feb271ee-936f-42c1-ae73-86b74af532a1'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:23.370760 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4ac8da1514e14849948adf4034874003', 'request_id': 'req-feb271ee-936f-42c1-ae73-86b74af532a1'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:23.377789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4ac8da1514e14849948adf4034874003', 'request_id': 'req-feb271ee-936f-42c1-ae73-86b74af532a1'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:23.389202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] The token cache is being invalidate because project 4ac8da1514e14849948adf4034874003 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:23.389484 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:23.390108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:23.391589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 895/1805] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:23 2022] DELETE /identity/v3/projects/4ac8da1514e14849948adf4034874003 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:23.395736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192ccfde-0b76-4c19-8032-6c05da3c99a8 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:23.405758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192ccfde-0b76-4c19-8032-6c05da3c99a8 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:23.459957 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd2c351-f7a7-45aa-9d93-a85bd5593e58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:23.460603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd2c351-f7a7-45aa-9d93-a85bd5593e58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:23.460881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd2c351-f7a7-45aa-9d93-a85bd5593e58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:23.461149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd2c351-f7a7-45aa-9d93-a85bd5593e58 admin admin] PATH_INFO: `/v3/users/be781cd70bc9415f813659b7b3783f8c` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:23.462278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd2c351-f7a7-45aa-9d93-a85bd5593e58 admin admin] RBAC: Authorizing `identity:delete_user(user_id=be781cd70bc9415f813659b7b3783f8c)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:23.464345 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd2c351-f7a7-45aa-9d93-a85bd5593e58 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:23.512572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bbd2c351-f7a7-45aa-9d93-a85bd5593e58 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be781cd70bc9415f813659b7b3783f8c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:23.513964 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bbd2c351-f7a7-45aa-9d93-a85bd5593e58 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be781cd70bc9415f813659b7b3783f8c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:23.531088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bbd2c351-f7a7-45aa-9d93-a85bd5593e58 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be781cd70bc9415f813659b7b3783f8c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:23.531795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bbd2c351-f7a7-45aa-9d93-a85bd5593e58 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be781cd70bc9415f813659b7b3783f8c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:23.534520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:23.553539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bbd2c351-f7a7-45aa-9d93-a85bd5593e58 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be781cd70bc9415f813659b7b3783f8c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:23.573153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 911/1806] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:23 2022] DELETE /identity/v3/users/be781cd70bc9415f813659b7b3783f8c => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:23.896533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:23.900523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:23.919321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb271ee-936f-42c1-ae73-86b74af532a1 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:23.995908 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:23.996570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:23.997055 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:23.997475 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8 admin admin] PATH_INFO: `/v3/projects/4d0b47a8cdb34e0daf711a2f1958c121` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:23.999422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4d0b47a8cdb34e0daf711a2f1958c121)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:24.001406 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:24.069128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4d0b47a8cdb34e0daf711a2f1958c121', 'request_id': 'req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:24.076845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4d0b47a8cdb34e0daf711a2f1958c121', 'request_id': 'req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:24.086118 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4d0b47a8cdb34e0daf711a2f1958c121', 'request_id': 'req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:24.101355 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8 admin admin] The token cache is being invalidate because project 4d0b47a8cdb34e0daf711a2f1958c121 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:24.101792 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:24.102569 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e1e44a8e-ec3b-4858-a78b-2067daaf5fd8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:24.104561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 896/1807] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:23 2022] DELETE /identity/v3/projects/4d0b47a8cdb34e0daf711a2f1958c121 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:24.110121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4757d48-6f47-4516-97f7-af02cf1e64d2 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:24.110629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4757d48-6f47-4516-97f7-af02cf1e64d2 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:24.110839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4757d48-6f47-4516-97f7-af02cf1e64d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:24.132542 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-c4757d48-6f47-4516-97f7-af02cf1e64d2 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:24.133865 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c4757d48-6f47-4516-97f7-af02cf1e64d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:24.195102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 912/1808] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:24.203801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5f2512-30ea-423b-8b6a-14777bed1a23 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:24.204095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5f2512-30ea-423b-8b6a-14777bed1a23 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:24.204519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5f2512-30ea-423b-8b6a-14777bed1a23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:24.225261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-bc5f2512-30ea-423b-8b6a-14777bed1a23 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:24.226458 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-bc5f2512-30ea-423b-8b6a-14777bed1a23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:24.234173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 897/1809] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:24.238426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4757d48-6f47-4516-97f7-af02cf1e64d2 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:24.248025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4757d48-6f47-4516-97f7-af02cf1e64d2 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:24.252567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7cd436-2d69-4a67-8917-fa2926a85944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:24.253158 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7cd436-2d69-4a67-8917-fa2926a85944 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:24.253436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7cd436-2d69-4a67-8917-fa2926a85944 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:24.253715 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7cd436-2d69-4a67-8917-fa2926a85944 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:24.254358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7cd436-2d69-4a67-8917-fa2926a85944 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:15:24.254665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7cd436-2d69-4a67-8917-fa2926a85944 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:24.256237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7cd436-2d69-4a67-8917-fa2926a85944 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:24.267979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 913/1810] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:15:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:15:24.272334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5f2512-30ea-423b-8b6a-14777bed1a23 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:24.282344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5f2512-30ea-423b-8b6a-14777bed1a23 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:24.288268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf889fa-c11e-40a0-b6cd-40682baa2cf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:24.288907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf889fa-c11e-40a0-b6cd-40682baa2cf9 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:24.289183 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf889fa-c11e-40a0-b6cd-40682baa2cf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:24.289451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf889fa-c11e-40a0-b6cd-40682baa2cf9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:24.290086 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf889fa-c11e-40a0-b6cd-40682baa2cf9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:24.292148 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf889fa-c11e-40a0-b6cd-40682baa2cf9 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:24.320066 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 898/1811] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:24 2022] POST /identity/v3/projects => generated 379 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:24.324630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7cd436-2d69-4a67-8917-fa2926a85944 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:24.347896 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7cd436-2d69-4a67-8917-fa2926a85944 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:24.438192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f942731-c16b-478f-a259-ea93c3077059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:24.438732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f942731-c16b-478f-a259-ea93c3077059 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:24.439007 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f942731-c16b-478f-a259-ea93c3077059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:24.439275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f942731-c16b-478f-a259-ea93c3077059 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:24.439908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f942731-c16b-478f-a259-ea93c3077059 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:24.441852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f942731-c16b-478f-a259-ea93c3077059 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:24.464798 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 914/1812] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:24 2022] POST /identity/v3/users => generated 290 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:24.469726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf889fa-c11e-40a0-b6cd-40682baa2cf9 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:24.475599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:24.483019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf889fa-c11e-40a0-b6cd-40682baa2cf9 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:24.494995 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf585ce-f497-4c1b-83b8-23995fa06302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:24.495517 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf585ce-f497-4c1b-83b8-23995fa06302 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:24.496837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf585ce-f497-4c1b-83b8-23995fa06302 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:24.497117 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf585ce-f497-4c1b-83b8-23995fa06302 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:24.498327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf585ce-f497-4c1b-83b8-23995fa06302 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:24.499978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf585ce-f497-4c1b-83b8-23995fa06302 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:24.516416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 899/1813] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:24 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:24.520624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f942731-c16b-478f-a259-ea93c3077059 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:24.530891 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f942731-c16b-478f-a259-ea93c3077059 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:24.535784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03789e9d-54a7-45b8-bdf9-c4379fb19a54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:24.536379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03789e9d-54a7-45b8-bdf9-c4379fb19a54 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:24.536658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03789e9d-54a7-45b8-bdf9-c4379fb19a54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:24.536926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03789e9d-54a7-45b8-bdf9-c4379fb19a54 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:24.537574 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03789e9d-54a7-45b8-bdf9-c4379fb19a54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:24.539904 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03789e9d-54a7-45b8-bdf9-c4379fb19a54 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:24.552397 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 915/1814] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:24.555220 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf585ce-f497-4c1b-83b8-23995fa06302 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:24.589358 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf585ce-f497-4c1b-83b8-23995fa06302 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:24.599878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6338cb50-ea12-4d4d-ba20-ff836cceca64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:24.600568 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6338cb50-ea12-4d4d-ba20-ff836cceca64 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:24.600831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6338cb50-ea12-4d4d-ba20-ff836cceca64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:24.601171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6338cb50-ea12-4d4d-ba20-ff836cceca64 admin admin] PATH_INFO: `/v3/projects/0ba56a928c5e4cc494b38b2ed40907ed/users/3bf57f0fe1244696b6a858e2cef0857b/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:24.639240 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6338cb50-ea12-4d4d-ba20-ff836cceca64 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ba56a928c5e4cc494b38b2ed40907ed, user_id=3bf57f0fe1244696b6a858e2cef0857b, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:24.642668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6338cb50-ea12-4d4d-ba20-ff836cceca64 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:24.676150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 900/1815] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:24 2022] PUT /identity/v3/projects/0ba56a928c5e4cc494b38b2ed40907ed/users/3bf57f0fe1244696b6a858e2cef0857b/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:24.687371 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2e3989-8478-42a2-bdaf-239cc3e25c49 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:24.688059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2e3989-8478-42a2-bdaf-239cc3e25c49 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:24.688385 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2e3989-8478-42a2-bdaf-239cc3e25c49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:24.740639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-0f2e3989-8478-42a2-bdaf-239cc3e25c49 None None] MFA Rules not processed for user `3bf57f0fe1244696b6a858e2cef0857b`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:24.742999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0f2e3989-8478-42a2-bdaf-239cc3e25c49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:24.803199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:24.852733 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 916/1816] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:24 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:24.857468 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6338cb50-ea12-4d4d-ba20-ff836cceca64 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:24.990040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6338cb50-ea12-4d4d-ba20-ff836cceca64 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:24.995024 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a40698-01f0-4a6b-ba61-b9e1add32e76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:24.995639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a40698-01f0-4a6b-ba61-b9e1add32e76 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:24.995976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a40698-01f0-4a6b-ba61-b9e1add32e76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:24.996259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a40698-01f0-4a6b-ba61-b9e1add32e76 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:24.997134 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a40698-01f0-4a6b-ba61-b9e1add32e76 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:24.999661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a40698-01f0-4a6b-ba61-b9e1add32e76 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:25.039070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 901/1817] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:24 2022] POST /identity/v3/projects => generated 379 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:25.043517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2e3989-8478-42a2-bdaf-239cc3e25c49 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:25.058768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2e3989-8478-42a2-bdaf-239cc3e25c49 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:25.141434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3d99f5-8f33-48f0-af6b-c098e4569c62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:25.142117 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3d99f5-8f33-48f0-af6b-c098e4569c62 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:25.142344 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3d99f5-8f33-48f0-af6b-c098e4569c62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:25.142635 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3d99f5-8f33-48f0-af6b-c098e4569c62 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:25.143289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3d99f5-8f33-48f0-af6b-c098e4569c62 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:25.147607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3d99f5-8f33-48f0-af6b-c098e4569c62 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:25.177683 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 917/1818] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:25 2022] POST /identity/v3/users => generated 296 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:25.181687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a40698-01f0-4a6b-ba61-b9e1add32e76 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:25.210812 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a40698-01f0-4a6b-ba61-b9e1add32e76 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:25.221784 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d26091b-9720-4775-a3f1-586f8cb6a3e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:25.223635 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d26091b-9720-4775-a3f1-586f8cb6a3e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:25.224141 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d26091b-9720-4775-a3f1-586f8cb6a3e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:25.224830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d26091b-9720-4775-a3f1-586f8cb6a3e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:25.227121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d26091b-9720-4775-a3f1-586f8cb6a3e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:25.230542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d26091b-9720-4775-a3f1-586f8cb6a3e0 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:25.245017 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 902/1819] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:25 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:25.250033 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3d99f5-8f33-48f0-af6b-c098e4569c62 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:25.262204 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3d99f5-8f33-48f0-af6b-c098e4569c62 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:25.269451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25c4c60-757c-4150-8790-82250483632d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:25.270472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25c4c60-757c-4150-8790-82250483632d admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:25.270759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25c4c60-757c-4150-8790-82250483632d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:25.271034 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25c4c60-757c-4150-8790-82250483632d admin admin] PATH_INFO: `/v3/projects/7a70b0d2d9d84d8c8895d765085cf440/users/6492a97c49394d53a9a057daac6252b3/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:25.289134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25c4c60-757c-4150-8790-82250483632d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a70b0d2d9d84d8c8895d765085cf440, user_id=6492a97c49394d53a9a057daac6252b3, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:25.291784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25c4c60-757c-4150-8790-82250483632d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:25.333173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 918/1820] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:25 2022] PUT /identity/v3/projects/7a70b0d2d9d84d8c8895d765085cf440/users/6492a97c49394d53a9a057daac6252b3/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:25.337471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d26091b-9720-4775-a3f1-586f8cb6a3e0 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:25.349014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d26091b-9720-4775-a3f1-586f8cb6a3e0 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:25.409698 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc2ef07-9807-4ed2-8fd6-25251bc735c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:25.410386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc2ef07-9807-4ed2-8fd6-25251bc735c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:25.410729 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc2ef07-9807-4ed2-8fd6-25251bc735c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:25.411207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc2ef07-9807-4ed2-8fd6-25251bc735c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:25.411791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc2ef07-9807-4ed2-8fd6-25251bc735c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:25.413487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc2ef07-9807-4ed2-8fd6-25251bc735c8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:25.427482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 903/1821] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:25 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:25.431914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25c4c60-757c-4150-8790-82250483632d admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:25.444403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25c4c60-757c-4150-8790-82250483632d admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:25.449479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5628760b-d6e2-4860-ac93-e5d5797deb5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:25.451003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5628760b-d6e2-4860-ac93-e5d5797deb5a admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:25.451419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5628760b-d6e2-4860-ac93-e5d5797deb5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:25.451844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5628760b-d6e2-4860-ac93-e5d5797deb5a admin admin] PATH_INFO: `/v3/projects/7a70b0d2d9d84d8c8895d765085cf440/users/6492a97c49394d53a9a057daac6252b3/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:25.453968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5628760b-d6e2-4860-ac93-e5d5797deb5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a70b0d2d9d84d8c8895d765085cf440, user_id=6492a97c49394d53a9a057daac6252b3, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:25.457170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5628760b-d6e2-4860-ac93-e5d5797deb5a admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:25.469544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 919/1822] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:25 2022] PUT /identity/v3/projects/7a70b0d2d9d84d8c8895d765085cf440/users/6492a97c49394d53a9a057daac6252b3/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:25.483585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d063c79-8563-4f5f-931d-67c44ece6268 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:25.483585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d063c79-8563-4f5f-931d-67c44ece6268 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:25.483585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d063c79-8563-4f5f-931d-67c44ece6268 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:25.512388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-1d063c79-8563-4f5f-931d-67c44ece6268 None None] MFA Rules not processed for user `6492a97c49394d53a9a057daac6252b3`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:25.514615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1d063c79-8563-4f5f-931d-67c44ece6268 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:25.546884 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:25.591207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 904/1823] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:25 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:25.602631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5628760b-d6e2-4860-ac93-e5d5797deb5a admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:25.604397 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5628760b-d6e2-4860-ac93-e5d5797deb5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:25.665417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5628760b-d6e2-4860-ac93-e5d5797deb5a admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:25.669014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28507d1-8859-4349-a800-2804fcf87c6a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:25.669659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28507d1-8859-4349-a800-2804fcf87c6a service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:25.669933 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28507d1-8859-4349-a800-2804fcf87c6a service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:25.670206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28507d1-8859-4349-a800-2804fcf87c6a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:25.679568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28507d1-8859-4349-a800-2804fcf87c6a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:25.681019 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28507d1-8859-4349-a800-2804fcf87c6a service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:25.691043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 920/1824] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:25 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:26.171115 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d063c79-8563-4f5f-931d-67c44ece6268 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:26.235292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d063c79-8563-4f5f-931d-67c44ece6268 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:26.239775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8c5e99-744e-4919-bd83-9fea10a2b906 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:26.240511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8c5e99-744e-4919-bd83-9fea10a2b906 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:26.240798 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8c5e99-744e-4919-bd83-9fea10a2b906 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:26.241074 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8c5e99-744e-4919-bd83-9fea10a2b906 admin admin] PATH_INFO: `/v3/users/3bf57f0fe1244696b6a858e2cef0857b` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:26.242242 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8c5e99-744e-4919-bd83-9fea10a2b906 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3bf57f0fe1244696b6a858e2cef0857b)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:26.244165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8c5e99-744e-4919-bd83-9fea10a2b906 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:26.290470 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-bc8c5e99-744e-4919-bd83-9fea10a2b906 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3bf57f0fe1244696b6a858e2cef0857b'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:26.291144 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-bc8c5e99-744e-4919-bd83-9fea10a2b906 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3bf57f0fe1244696b6a858e2cef0857b'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:26.291750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-bc8c5e99-744e-4919-bd83-9fea10a2b906 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3bf57f0fe1244696b6a858e2cef0857b'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:26.300461 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-bc8c5e99-744e-4919-bd83-9fea10a2b906 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3bf57f0fe1244696b6a858e2cef0857b'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:26.312951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-bc8c5e99-744e-4919-bd83-9fea10a2b906 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3bf57f0fe1244696b6a858e2cef0857b'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:26.326312 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 905/1825] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:26 2022] DELETE /identity/v3/users/3bf57f0fe1244696b6a858e2cef0857b => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:26.331069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185e7497-e47f-4afc-bda7-5bf884f4fd29 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:26.331546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185e7497-e47f-4afc-bda7-5bf884f4fd29 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:26.331649 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185e7497-e47f-4afc-bda7-5bf884f4fd29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:26.351567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-185e7497-e47f-4afc-bda7-5bf884f4fd29 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:26.352815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-185e7497-e47f-4afc-bda7-5bf884f4fd29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:26.374558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:26.419734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 921/1826] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:26.430416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8c5e99-744e-4919-bd83-9fea10a2b906 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:26.433020 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-bc8c5e99-744e-4919-bd83-9fea10a2b906 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:26.492633 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:26.507288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8c5e99-744e-4919-bd83-9fea10a2b906 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:26.521807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baf68e1-2674-4500-a19f-34f637946300 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:26.524022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baf68e1-2674-4500-a19f-34f637946300 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:26.524526 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baf68e1-2674-4500-a19f-34f637946300 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:26.524941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baf68e1-2674-4500-a19f-34f637946300 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:26.536290 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2baf68e1-2674-4500-a19f-34f637946300 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:26.537754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2baf68e1-2674-4500-a19f-34f637946300 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:26.549325 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 906/1827] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:26.891910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185e7497-e47f-4afc-bda7-5bf884f4fd29 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:26.893371 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-185e7497-e47f-4afc-bda7-5bf884f4fd29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:26.905520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185e7497-e47f-4afc-bda7-5bf884f4fd29 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:26.910263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:26.910901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:26.911182 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:26.911459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] PATH_INFO: `/v3/projects/0ba56a928c5e4cc494b38b2ed40907ed` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:26.912628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ba56a928c5e4cc494b38b2ed40907ed)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:26.914538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:26.961789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ba56a928c5e4cc494b38b2ed40907ed', 'request_id': 'req-bc735c43-7b42-449e-8824-6bf3c813a6b7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:26.967296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ba56a928c5e4cc494b38b2ed40907ed', 'request_id': 'req-bc735c43-7b42-449e-8824-6bf3c813a6b7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:26.974955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ba56a928c5e4cc494b38b2ed40907ed', 'request_id': 'req-bc735c43-7b42-449e-8824-6bf3c813a6b7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:26.985771 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] The token cache is being invalidate because project 0ba56a928c5e4cc494b38b2ed40907ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:26.986055 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:26.986696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:26.988195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 922/1828] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:26 2022] DELETE /identity/v3/projects/0ba56a928c5e4cc494b38b2ed40907ed => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:26.992291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baf68e1-2674-4500-a19f-34f637946300 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:27.004043 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baf68e1-2674-4500-a19f-34f637946300 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:27.059739 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7288e6-ed38-4e35-b2e7-e38afe441012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:27.060933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7288e6-ed38-4e35-b2e7-e38afe441012 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:27.061208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7288e6-ed38-4e35-b2e7-e38afe441012 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:27.061476 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7288e6-ed38-4e35-b2e7-e38afe441012 admin admin] PATH_INFO: `/v3/users/6492a97c49394d53a9a057daac6252b3` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:27.062579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7288e6-ed38-4e35-b2e7-e38afe441012 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6492a97c49394d53a9a057daac6252b3)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:27.064516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7288e6-ed38-4e35-b2e7-e38afe441012 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:27.112933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ab7288e6-ed38-4e35-b2e7-e38afe441012 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6492a97c49394d53a9a057daac6252b3'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:27.113588 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ab7288e6-ed38-4e35-b2e7-e38afe441012 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6492a97c49394d53a9a057daac6252b3'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:27.114255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ab7288e6-ed38-4e35-b2e7-e38afe441012 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6492a97c49394d53a9a057daac6252b3'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:27.121563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ab7288e6-ed38-4e35-b2e7-e38afe441012 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6492a97c49394d53a9a057daac6252b3'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:27.133962 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ab7288e6-ed38-4e35-b2e7-e38afe441012 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6492a97c49394d53a9a057daac6252b3'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:27.146334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 907/1829] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:26 2022] DELETE /identity/v3/users/6492a97c49394d53a9a057daac6252b3 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:27.499266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:27.500834 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:27.515235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc735c43-7b42-449e-8824-6bf3c813a6b7 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:27.579646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:27.580283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:27.580616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:27.580892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c admin admin] PATH_INFO: `/v3/projects/7a70b0d2d9d84d8c8895d765085cf440` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:27.582046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a70b0d2d9d84d8c8895d765085cf440)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:27.584542 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:27.642346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a70b0d2d9d84d8c8895d765085cf440', 'request_id': 'req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:27.644538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:27.650200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a70b0d2d9d84d8c8895d765085cf440', 'request_id': 'req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:27.661977 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a70b0d2d9d84d8c8895d765085cf440', 'request_id': 'req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:27.677781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c admin admin] The token cache is being invalidate because project 7a70b0d2d9d84d8c8895d765085cf440 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:27.677781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:27.678129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fbb308ee-686b-4a7e-96c0-0ee0f5591c3c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:27.679689 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 923/1830] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:27 2022] DELETE /identity/v3/projects/7a70b0d2d9d84d8c8895d765085cf440 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:27.684894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3545c8ae-f45e-471f-b609-d11916ee0f58 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:27.685544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3545c8ae-f45e-471f-b609-d11916ee0f58 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:27.685786 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3545c8ae-f45e-471f-b609-d11916ee0f58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:27.709270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3545c8ae-f45e-471f-b609-d11916ee0f58 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:27.710500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3545c8ae-f45e-471f-b609-d11916ee0f58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:27.772442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 908/1831] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:27.780511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c997d8-6601-4457-bc9d-8de22ec6693a None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:27.780794 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c997d8-6601-4457-bc9d-8de22ec6693a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:27.781067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c997d8-6601-4457-bc9d-8de22ec6693a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:27.801917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-35c997d8-6601-4457-bc9d-8de22ec6693a None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:27.803127 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-35c997d8-6601-4457-bc9d-8de22ec6693a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:27.811547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 924/1832] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:27.815877 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3545c8ae-f45e-471f-b609-d11916ee0f58 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:27.821876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:27.826121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3545c8ae-f45e-471f-b609-d11916ee0f58 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:27.832366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1d915e-bb44-4f23-9a92-8a01ec4dd12a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:27.833281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1d915e-bb44-4f23-9a92-8a01ec4dd12a admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:27.833654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1d915e-bb44-4f23-9a92-8a01ec4dd12a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:27.834084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1d915e-bb44-4f23-9a92-8a01ec4dd12a admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:27.834797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1d915e-bb44-4f23-9a92-8a01ec4dd12a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:15:27.835320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1d915e-bb44-4f23-9a92-8a01ec4dd12a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:27.836986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1d915e-bb44-4f23-9a92-8a01ec4dd12a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:27.849696 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 909/1833] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:15:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:15:27.854217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c997d8-6601-4457-bc9d-8de22ec6693a None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:27.864395 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c997d8-6601-4457-bc9d-8de22ec6693a None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:27.869232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049f93a4-3915-4dac-b3e5-d162e688d5a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:27.869835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049f93a4-3915-4dac-b3e5-d162e688d5a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:27.870112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049f93a4-3915-4dac-b3e5-d162e688d5a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:27.870387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049f93a4-3915-4dac-b3e5-d162e688d5a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:27.871061 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049f93a4-3915-4dac-b3e5-d162e688d5a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:27.873100 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049f93a4-3915-4dac-b3e5-d162e688d5a6 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:27.903252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 925/1834] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:27 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:15:27.905487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1d915e-bb44-4f23-9a92-8a01ec4dd12a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:27.917370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1d915e-bb44-4f23-9a92-8a01ec4dd12a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:27.976279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d552414-9386-4f24-8ff7-380e8b1db45d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:27.977175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d552414-9386-4f24-8ff7-380e8b1db45d admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:27.977463 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d552414-9386-4f24-8ff7-380e8b1db45d admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:27.977750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d552414-9386-4f24-8ff7-380e8b1db45d admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:27.978346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d552414-9386-4f24-8ff7-380e8b1db45d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:27.980253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d552414-9386-4f24-8ff7-380e8b1db45d admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:28.000717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 910/1835] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:27 2022] POST /identity/v3/users => generated 293 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:28.004853 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049f93a4-3915-4dac-b3e5-d162e688d5a6 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:28.015516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049f93a4-3915-4dac-b3e5-d162e688d5a6 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:28.020328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61865c27-415c-44a6-8963-c14c68fb1cdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:28.020972 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61865c27-415c-44a6-8963-c14c68fb1cdd admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:28.021266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61865c27-415c-44a6-8963-c14c68fb1cdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:28.021520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61865c27-415c-44a6-8963-c14c68fb1cdd admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:28.022190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61865c27-415c-44a6-8963-c14c68fb1cdd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:28.023802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61865c27-415c-44a6-8963-c14c68fb1cdd admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:28.034363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 926/1836] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:28.038233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d552414-9386-4f24-8ff7-380e8b1db45d admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:28.047998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d552414-9386-4f24-8ff7-380e8b1db45d admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:28.052823 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186771a7-c65d-4701-8e0c-943a8e9075e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:28.053411 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186771a7-c65d-4701-8e0c-943a8e9075e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:28.053687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186771a7-c65d-4701-8e0c-943a8e9075e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:28.053955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186771a7-c65d-4701-8e0c-943a8e9075e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:28.054642 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186771a7-c65d-4701-8e0c-943a8e9075e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:28.056264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186771a7-c65d-4701-8e0c-943a8e9075e8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:28.066641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 911/1837] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:28 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:28.070581 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61865c27-415c-44a6-8963-c14c68fb1cdd admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:28.080937 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61865c27-415c-44a6-8963-c14c68fb1cdd admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:28.085450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae90ffc3-48b4-4520-a118-0048ed90fc16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:28.086076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae90ffc3-48b4-4520-a118-0048ed90fc16 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:28.086355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae90ffc3-48b4-4520-a118-0048ed90fc16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:28.086631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae90ffc3-48b4-4520-a118-0048ed90fc16 admin admin] PATH_INFO: `/v3/projects/c3e76fef546d4152bc0d60dfc4dc4304/users/7fe58b0d4f6646ce9209b5d323476076/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:28.101877 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae90ffc3-48b4-4520-a118-0048ed90fc16 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3e76fef546d4152bc0d60dfc4dc4304, user_id=7fe58b0d4f6646ce9209b5d323476076, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:28.104550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae90ffc3-48b4-4520-a118-0048ed90fc16 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:28.116914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 927/1838] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:28 2022] PUT /identity/v3/projects/c3e76fef546d4152bc0d60dfc4dc4304/users/7fe58b0d4f6646ce9209b5d323476076/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:15:28.127905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eccc53a-869d-4b98-b691-984ddc3e16d4 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:28.128187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eccc53a-869d-4b98-b691-984ddc3e16d4 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:28.128488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eccc53a-869d-4b98-b691-984ddc3e16d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:28.155944 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-9eccc53a-869d-4b98-b691-984ddc3e16d4 None None] MFA Rules not processed for user `7fe58b0d4f6646ce9209b5d323476076`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:28.157166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9eccc53a-869d-4b98-b691-984ddc3e16d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:28.228852 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 912/1839] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:28 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:28.233239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae90ffc3-48b4-4520-a118-0048ed90fc16 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:28.293522 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae90ffc3-48b4-4520-a118-0048ed90fc16 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:28.298160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e59e3e-7f57-45b8-9f2f-f1f2e789ff1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:28.298762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e59e3e-7f57-45b8-9f2f-f1f2e789ff1a admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:28.299046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e59e3e-7f57-45b8-9f2f-f1f2e789ff1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:28.299324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e59e3e-7f57-45b8-9f2f-f1f2e789ff1a admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:28.299959 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e59e3e-7f57-45b8-9f2f-f1f2e789ff1a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:28.301896 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e59e3e-7f57-45b8-9f2f-f1f2e789ff1a admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:28.329091 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 928/1840] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:28 2022] POST /identity/v3/projects => generated 383 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:28.333173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eccc53a-869d-4b98-b691-984ddc3e16d4 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:28.344225 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eccc53a-869d-4b98-b691-984ddc3e16d4 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:28.407090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4deb21cd-9aed-4749-a914-9e565febd55b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:28.407707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deb21cd-9aed-4749-a914-9e565febd55b admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:28.407979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deb21cd-9aed-4749-a914-9e565febd55b admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:28.408247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deb21cd-9aed-4749-a914-9e565febd55b admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:28.408922 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4deb21cd-9aed-4749-a914-9e565febd55b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:28.410864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4deb21cd-9aed-4749-a914-9e565febd55b admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:28.419570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:28.437804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 913/1841] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:28 2022] POST /identity/v3/users => generated 298 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:15:28.441775 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e59e3e-7f57-45b8-9f2f-f1f2e789ff1a admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:28.453405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e59e3e-7f57-45b8-9f2f-f1f2e789ff1a admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:28.458320 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f950e87-eddd-48fd-a534-41a876fb5960 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:28.459001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f950e87-eddd-48fd-a534-41a876fb5960 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:28.459289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f950e87-eddd-48fd-a534-41a876fb5960 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:28.459567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f950e87-eddd-48fd-a534-41a876fb5960 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:28.460222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f950e87-eddd-48fd-a534-41a876fb5960 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:28.461838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f950e87-eddd-48fd-a534-41a876fb5960 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:28.472906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 929/1842] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:28 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:15:28.477161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4deb21cd-9aed-4749-a914-9e565febd55b admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:28.487654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4deb21cd-9aed-4749-a914-9e565febd55b admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:28.494562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbe444b-9e0f-439f-9749-1e0c7e481dc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:28.498329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbe444b-9e0f-439f-9749-1e0c7e481dc8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:28.498835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbe444b-9e0f-439f-9749-1e0c7e481dc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:28.499341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbe444b-9e0f-439f-9749-1e0c7e481dc8 admin admin] PATH_INFO: `/v3/projects/15694025b75d48fea0b2ab98a85cfb35/users/b87f7c8626eb4ec2838eb721ecfc6289/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:28.520377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbe444b-9e0f-439f-9749-1e0c7e481dc8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15694025b75d48fea0b2ab98a85cfb35, user_id=b87f7c8626eb4ec2838eb721ecfc6289, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:28.528253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbe444b-9e0f-439f-9749-1e0c7e481dc8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:28.545703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 914/1843] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:28 2022] PUT /identity/v3/projects/15694025b75d48fea0b2ab98a85cfb35/users/b87f7c8626eb4ec2838eb721ecfc6289/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:28.549796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f950e87-eddd-48fd-a534-41a876fb5960 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:28.562451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f950e87-eddd-48fd-a534-41a876fb5960 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:28.624086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:28.643028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc2b754-9f58-45c6-abc9-9a77d9d55710 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:28.644108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc2b754-9f58-45c6-abc9-9a77d9d55710 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:28.644385 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc2b754-9f58-45c6-abc9-9a77d9d55710 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:28.644840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc2b754-9f58-45c6-abc9-9a77d9d55710 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:28.645800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc2b754-9f58-45c6-abc9-9a77d9d55710 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:28.648237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc2b754-9f58-45c6-abc9-9a77d9d55710 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:28.661327 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 930/1844] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:28 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:28.665400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbe444b-9e0f-439f-9749-1e0c7e481dc8 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:28.677300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbe444b-9e0f-439f-9749-1e0c7e481dc8 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:28.682628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822c197e-1918-40e6-bd1c-922ed87edd62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:28.683430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822c197e-1918-40e6-bd1c-922ed87edd62 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:28.683786 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822c197e-1918-40e6-bd1c-922ed87edd62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:28.684129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822c197e-1918-40e6-bd1c-922ed87edd62 admin admin] PATH_INFO: `/v3/projects/15694025b75d48fea0b2ab98a85cfb35/users/b87f7c8626eb4ec2838eb721ecfc6289/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:28.686885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822c197e-1918-40e6-bd1c-922ed87edd62 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15694025b75d48fea0b2ab98a85cfb35, user_id=b87f7c8626eb4ec2838eb721ecfc6289, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:28.689887 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822c197e-1918-40e6-bd1c-922ed87edd62 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:28.724284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 915/1845] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:28 2022] PUT /identity/v3/projects/15694025b75d48fea0b2ab98a85cfb35/users/b87f7c8626eb4ec2838eb721ecfc6289/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:28.725759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96a9903-9890-44a2-a209-a51223397265 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:28.726285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96a9903-9890-44a2-a209-a51223397265 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:28.726672 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96a9903-9890-44a2-a209-a51223397265 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:28.787289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-d96a9903-9890-44a2-a209-a51223397265 None None] MFA Rules not processed for user `b87f7c8626eb4ec2838eb721ecfc6289`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:28.788631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d96a9903-9890-44a2-a209-a51223397265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:28.883545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 931/1846] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:28 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:28.895366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822c197e-1918-40e6-bd1c-922ed87edd62 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:28.897569 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-822c197e-1918-40e6-bd1c-922ed87edd62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:28.968717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822c197e-1918-40e6-bd1c-922ed87edd62 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:28.973571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1af81e-2a31-4bae-9430-dd3c4292b9ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:28.974180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1af81e-2a31-4bae-9430-dd3c4292b9ed service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:28.974462 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1af81e-2a31-4bae-9430-dd3c4292b9ed service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:28.974727 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1af81e-2a31-4bae-9430-dd3c4292b9ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:29.041598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1af81e-2a31-4bae-9430-dd3c4292b9ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:29.043150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1af81e-2a31-4bae-9430-dd3c4292b9ed service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:29.054687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 916/1847] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:28 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:30.931002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96a9903-9890-44a2-a209-a51223397265 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:30.933012 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d96a9903-9890-44a2-a209-a51223397265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:31.022519 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96a9903-9890-44a2-a209-a51223397265 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:31.027762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efe5285-f8f4-4e4a-b33d-b09948e8528c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:31.028444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efe5285-f8f4-4e4a-b33d-b09948e8528c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:31.029056 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efe5285-f8f4-4e4a-b33d-b09948e8528c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:31.029342 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efe5285-f8f4-4e4a-b33d-b09948e8528c admin admin] PATH_INFO: `/v3/users/772a9a9fda4c446fa3d08eaae1ab9c82` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:31.030825 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efe5285-f8f4-4e4a-b33d-b09948e8528c admin admin] RBAC: Authorizing `identity:delete_user(user_id=772a9a9fda4c446fa3d08eaae1ab9c82)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:31.033190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efe5285-f8f4-4e4a-b33d-b09948e8528c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:31.090767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-3efe5285-f8f4-4e4a-b33d-b09948e8528c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '772a9a9fda4c446fa3d08eaae1ab9c82'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:31.092281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-3efe5285-f8f4-4e4a-b33d-b09948e8528c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '772a9a9fda4c446fa3d08eaae1ab9c82'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:31.101131 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-3efe5285-f8f4-4e4a-b33d-b09948e8528c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '772a9a9fda4c446fa3d08eaae1ab9c82'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:31.102275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-3efe5285-f8f4-4e4a-b33d-b09948e8528c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '772a9a9fda4c446fa3d08eaae1ab9c82'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:31.120417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-3efe5285-f8f4-4e4a-b33d-b09948e8528c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '772a9a9fda4c446fa3d08eaae1ab9c82'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:31.122571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:31.149199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 932/1848] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:30 2022] DELETE /identity/v3/users/772a9a9fda4c446fa3d08eaae1ab9c82 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:31.154657 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703bb283-a9f1-465c-aef3-2acc4b79ef9f None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:31.155875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703bb283-a9f1-465c-aef3-2acc4b79ef9f None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:31.156424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703bb283-a9f1-465c-aef3-2acc4b79ef9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:31.179795 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-703bb283-a9f1-465c-aef3-2acc4b79ef9f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:31.180953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-703bb283-a9f1-465c-aef3-2acc4b79ef9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:31.267473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 917/1849] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:31.268606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efe5285-f8f4-4e4a-b33d-b09948e8528c admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:31.270037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-3efe5285-f8f4-4e4a-b33d-b09948e8528c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:31.334981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efe5285-f8f4-4e4a-b33d-b09948e8528c admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:31.340273 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603eeb6b-7f6b-469a-af7c-ed12842b48f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:31.340782 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603eeb6b-7f6b-469a-af7c-ed12842b48f2 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:31.341062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603eeb6b-7f6b-469a-af7c-ed12842b48f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:31.341375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603eeb6b-7f6b-469a-af7c-ed12842b48f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:31.352338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603eeb6b-7f6b-469a-af7c-ed12842b48f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:31.353754 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603eeb6b-7f6b-469a-af7c-ed12842b48f2 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:31.363118 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 933/1850] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:31.802124 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703bb283-a9f1-465c-aef3-2acc4b79ef9f None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:31.804232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-703bb283-a9f1-465c-aef3-2acc4b79ef9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:31.817315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703bb283-a9f1-465c-aef3-2acc4b79ef9f None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:31.822016 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:31.822650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:31.822938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:31.823199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] PATH_INFO: `/v3/projects/b287e1f53b0749c5904b1e7ef25b3b46` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:31.824491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b287e1f53b0749c5904b1e7ef25b3b46)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:31.826487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:31.828243 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:31.889021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b287e1f53b0749c5904b1e7ef25b3b46', 'request_id': 'req-dac693a7-3dc4-4e5f-b55c-8b821218d802'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:31.895904 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b287e1f53b0749c5904b1e7ef25b3b46', 'request_id': 'req-dac693a7-3dc4-4e5f-b55c-8b821218d802'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:31.905861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b287e1f53b0749c5904b1e7ef25b3b46', 'request_id': 'req-dac693a7-3dc4-4e5f-b55c-8b821218d802'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:31.924237 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] The token cache is being invalidate because project b287e1f53b0749c5904b1e7ef25b3b46 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:31.924597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:31.925284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:31.926792 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 918/1851] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:31 2022] DELETE /identity/v3/projects/b287e1f53b0749c5904b1e7ef25b3b46 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:31.930717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603eeb6b-7f6b-469a-af7c-ed12842b48f2 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:31.940656 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603eeb6b-7f6b-469a-af7c-ed12842b48f2 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:32.007800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e52769c-0569-456d-ac14-e917499de36a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:32.008479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e52769c-0569-456d-ac14-e917499de36a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:32.008763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e52769c-0569-456d-ac14-e917499de36a admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:32.009039 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e52769c-0569-456d-ac14-e917499de36a admin admin] PATH_INFO: `/v3/users/3716a572b8a84368bdbe0163900048f2` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:32.010311 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e52769c-0569-456d-ac14-e917499de36a admin admin] RBAC: Authorizing `identity:delete_user(user_id=3716a572b8a84368bdbe0163900048f2)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:32.014505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e52769c-0569-456d-ac14-e917499de36a admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:32.069832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-9e52769c-0569-456d-ac14-e917499de36a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3716a572b8a84368bdbe0163900048f2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:32.070502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-9e52769c-0569-456d-ac14-e917499de36a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3716a572b8a84368bdbe0163900048f2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:32.078847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-9e52769c-0569-456d-ac14-e917499de36a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3716a572b8a84368bdbe0163900048f2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:32.079867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-9e52769c-0569-456d-ac14-e917499de36a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3716a572b8a84368bdbe0163900048f2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:32.099343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-9e52769c-0569-456d-ac14-e917499de36a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3716a572b8a84368bdbe0163900048f2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:32.112690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 934/1852] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:31 2022] DELETE /identity/v3/users/3716a572b8a84368bdbe0163900048f2 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:32.399855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:32.401399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:32.414552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac693a7-3dc4-4e5f-b55c-8b821218d802 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:32.488040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194a0b39-7ea6-4076-822a-769dbb106fd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:32.488727 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194a0b39-7ea6-4076-822a-769dbb106fd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:32.489145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194a0b39-7ea6-4076-822a-769dbb106fd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:32.489443 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194a0b39-7ea6-4076-822a-769dbb106fd0 admin admin] PATH_INFO: `/v3/projects/78ab99f10fb64157818f4d5444f2a7bf` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:32.491283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194a0b39-7ea6-4076-822a-769dbb106fd0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=78ab99f10fb64157818f4d5444f2a7bf)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:32.493585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194a0b39-7ea6-4076-822a-769dbb106fd0 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:32.553698 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-194a0b39-7ea6-4076-822a-769dbb106fd0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78ab99f10fb64157818f4d5444f2a7bf', 'request_id': 'req-194a0b39-7ea6-4076-822a-769dbb106fd0'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:32.559605 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-194a0b39-7ea6-4076-822a-769dbb106fd0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78ab99f10fb64157818f4d5444f2a7bf', 'request_id': 'req-194a0b39-7ea6-4076-822a-769dbb106fd0'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:32.569597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-194a0b39-7ea6-4076-822a-769dbb106fd0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78ab99f10fb64157818f4d5444f2a7bf', 'request_id': 'req-194a0b39-7ea6-4076-822a-769dbb106fd0'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:32.583682 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-194a0b39-7ea6-4076-822a-769dbb106fd0 admin admin] The token cache is being invalidate because project 78ab99f10fb64157818f4d5444f2a7bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:32.583981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-194a0b39-7ea6-4076-822a-769dbb106fd0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:32.585175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-194a0b39-7ea6-4076-822a-769dbb106fd0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:32.586847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 919/1853] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:32 2022] DELETE /identity/v3/projects/78ab99f10fb64157818f4d5444f2a7bf => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:32.592171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22ce0af-d0a0-4d38-a7c1-57959891e91a None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:32.592491 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22ce0af-d0a0-4d38-a7c1-57959891e91a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:32.592765 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22ce0af-d0a0-4d38-a7c1-57959891e91a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:32.615103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-a22ce0af-d0a0-4d38-a7c1-57959891e91a None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:32.616415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a22ce0af-d0a0-4d38-a7c1-57959891e91a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:32.626970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:INSERT=2,keystone:DELETE=11,keystone:SELECT=87 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:32.690048 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 935/1854] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:32.697457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ddeb04-b095-4c81-ad82-b6637d2db4cc None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:32.697746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ddeb04-b095-4c81-ad82-b6637d2db4cc None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:32.698033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ddeb04-b095-4c81-ad82-b6637d2db4cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:32.717937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-25ddeb04-b095-4c81-ad82-b6637d2db4cc None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:32.719402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-25ddeb04-b095-4c81-ad82-b6637d2db4cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:32.727640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 920/1855] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:32.732623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22ce0af-d0a0-4d38-a7c1-57959891e91a None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:32.742442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22ce0af-d0a0-4d38-a7c1-57959891e91a None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:32.747142 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f8462f-8d75-488e-9dca-7b37d289271c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:32.748351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f8462f-8d75-488e-9dca-7b37d289271c admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:32.748622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f8462f-8d75-488e-9dca-7b37d289271c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:32.749037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f8462f-8d75-488e-9dca-7b37d289271c admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:32.749688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f8462f-8d75-488e-9dca-7b37d289271c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:15:32.749994 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f8462f-8d75-488e-9dca-7b37d289271c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:32.751582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f8462f-8d75-488e-9dca-7b37d289271c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:32.763622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 936/1856] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:15:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:15:32.767899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ddeb04-b095-4c81-ad82-b6637d2db4cc None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:32.777296 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ddeb04-b095-4c81-ad82-b6637d2db4cc None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:32.782456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cfaf39-0591-440e-bfa3-b630d8b65673 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:32.783067 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cfaf39-0591-440e-bfa3-b630d8b65673 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:32.783345 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cfaf39-0591-440e-bfa3-b630d8b65673 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:32.783617 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cfaf39-0591-440e-bfa3-b630d8b65673 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:32.784244 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cfaf39-0591-440e-bfa3-b630d8b65673 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:32.786178 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cfaf39-0591-440e-bfa3-b630d8b65673 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:32.797620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:32.815067 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 921/1857] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:32 2022] POST /identity/v3/projects => generated 359 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:32.818919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f8462f-8d75-488e-9dca-7b37d289271c admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:32.828941 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f8462f-8d75-488e-9dca-7b37d289271c admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:32.889932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60e078f-e1d1-4156-9914-990978b4533b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:32.890474 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60e078f-e1d1-4156-9914-990978b4533b admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:32.890757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60e078f-e1d1-4156-9914-990978b4533b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:32.891033 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60e078f-e1d1-4156-9914-990978b4533b admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:32.891675 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60e078f-e1d1-4156-9914-990978b4533b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:32.894020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60e078f-e1d1-4156-9914-990978b4533b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:32.921897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 937/1858] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:32 2022] POST /identity/v3/users => generated 280 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:32.923068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cfaf39-0591-440e-bfa3-b630d8b65673 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:32.950540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cfaf39-0591-440e-bfa3-b630d8b65673 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:32.958399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec4059e-a451-4451-ad24-10173fce667e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:32.959029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec4059e-a451-4451-ad24-10173fce667e admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:32.959336 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec4059e-a451-4451-ad24-10173fce667e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:32.959615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec4059e-a451-4451-ad24-10173fce667e admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:32.960272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec4059e-a451-4451-ad24-10173fce667e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:32.961912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec4059e-a451-4451-ad24-10173fce667e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:32.975467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 922/1859] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:32 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:15:32.979550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60e078f-e1d1-4156-9914-990978b4533b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:32.994582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60e078f-e1d1-4156-9914-990978b4533b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:33.000483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fa6da1-62c3-424f-96de-c3adad71cf14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:33.001097 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fa6da1-62c3-424f-96de-c3adad71cf14 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:33.001381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fa6da1-62c3-424f-96de-c3adad71cf14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:33.001657 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fa6da1-62c3-424f-96de-c3adad71cf14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:33.002314 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fa6da1-62c3-424f-96de-c3adad71cf14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:33.003895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fa6da1-62c3-424f-96de-c3adad71cf14 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:33.015584 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 938/1860] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:32 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:33.019893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec4059e-a451-4451-ad24-10173fce667e admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:33.030200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec4059e-a451-4451-ad24-10173fce667e admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:33.034963 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257ea9ec-861d-47be-82b4-2a56141cf23a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:33.035616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257ea9ec-861d-47be-82b4-2a56141cf23a admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:33.035894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257ea9ec-861d-47be-82b4-2a56141cf23a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:33.036184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257ea9ec-861d-47be-82b4-2a56141cf23a admin admin] PATH_INFO: `/v3/projects/927bffce9c494f559693b747ea737c74/users/bd5cccbef8294d67966bfe1b0979c9b4/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:33.051497 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257ea9ec-861d-47be-82b4-2a56141cf23a admin admin] RBAC: Authorizing `identity:create_grant(project_id=927bffce9c494f559693b747ea737c74, user_id=bd5cccbef8294d67966bfe1b0979c9b4, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:33.054192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257ea9ec-861d-47be-82b4-2a56141cf23a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:33.069011 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 923/1861] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:33 2022] PUT /identity/v3/projects/927bffce9c494f559693b747ea737c74/users/bd5cccbef8294d67966bfe1b0979c9b4/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:33.080956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b66b830-8da1-4ed3-a64c-3acc05a5e197 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:33.081244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b66b830-8da1-4ed3-a64c-3acc05a5e197 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:33.081516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b66b830-8da1-4ed3-a64c-3acc05a5e197 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:33.110294 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-0b66b830-8da1-4ed3-a64c-3acc05a5e197 None None] MFA Rules not processed for user `bd5cccbef8294d67966bfe1b0979c9b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:33.111501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0b66b830-8da1-4ed3-a64c-3acc05a5e197 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:33.183839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 939/1862] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:33 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:33.222278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257ea9ec-861d-47be-82b4-2a56141cf23a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:33.224420 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-257ea9ec-861d-47be-82b4-2a56141cf23a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:33.302100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257ea9ec-861d-47be-82b4-2a56141cf23a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:33.307166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb3bab7-28d0-4bd8-8ce4-b35ccdcf59c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:33.307893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb3bab7-28d0-4bd8-8ce4-b35ccdcf59c3 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:33.308246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb3bab7-28d0-4bd8-8ce4-b35ccdcf59c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:33.308624 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb3bab7-28d0-4bd8-8ce4-b35ccdcf59c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:33.320117 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb3bab7-28d0-4bd8-8ce4-b35ccdcf59c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:33.321649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb3bab7-28d0-4bd8-8ce4-b35ccdcf59c3 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:33.331439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 924/1863] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:33 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:33.348349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b66b830-8da1-4ed3-a64c-3acc05a5e197 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:33.357818 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:33.410264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b66b830-8da1-4ed3-a64c-3acc05a5e197 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:33.414775 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532ccd0c-12e3-4798-afef-58d94673b6d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:33.415393 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532ccd0c-12e3-4798-afef-58d94673b6d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:33.415674 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532ccd0c-12e3-4798-afef-58d94673b6d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:33.415948 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532ccd0c-12e3-4798-afef-58d94673b6d3 admin admin] PATH_INFO: `/v3/users/bd5cccbef8294d67966bfe1b0979c9b4` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:33.417102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532ccd0c-12e3-4798-afef-58d94673b6d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bd5cccbef8294d67966bfe1b0979c9b4)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:33.419031 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532ccd0c-12e3-4798-afef-58d94673b6d3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:33.463547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-532ccd0c-12e3-4798-afef-58d94673b6d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bd5cccbef8294d67966bfe1b0979c9b4'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:33.464127 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-532ccd0c-12e3-4798-afef-58d94673b6d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bd5cccbef8294d67966bfe1b0979c9b4'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:33.471833 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-532ccd0c-12e3-4798-afef-58d94673b6d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bd5cccbef8294d67966bfe1b0979c9b4'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:33.472499 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-532ccd0c-12e3-4798-afef-58d94673b6d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bd5cccbef8294d67966bfe1b0979c9b4'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:33.484188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-532ccd0c-12e3-4798-afef-58d94673b6d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bd5cccbef8294d67966bfe1b0979c9b4'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:33.497062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 940/1864] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:33 2022] DELETE /identity/v3/users/bd5cccbef8294d67966bfe1b0979c9b4 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:33.502367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e9fc3b-7be4-40cc-857c-e0a32995ec2d None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:33.502822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e9fc3b-7be4-40cc-857c-e0a32995ec2d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:33.502912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e9fc3b-7be4-40cc-857c-e0a32995ec2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:33.521844 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-82e9fc3b-7be4-40cc-857c-e0a32995ec2d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:33.523035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-82e9fc3b-7be4-40cc-857c-e0a32995ec2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:33.579694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 925/1865] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:33.589286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532ccd0c-12e3-4798-afef-58d94673b6d3 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:33.590671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-532ccd0c-12e3-4798-afef-58d94673b6d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:33.649415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532ccd0c-12e3-4798-afef-58d94673b6d3 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:33.652990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff1f6ff-7fd7-46e5-8362-0915bf8ced1f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:33.654086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff1f6ff-7fd7-46e5-8362-0915bf8ced1f service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:33.654378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff1f6ff-7fd7-46e5-8362-0915bf8ced1f service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:33.654645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff1f6ff-7fd7-46e5-8362-0915bf8ced1f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:33.663593 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff1f6ff-7fd7-46e5-8362-0915bf8ced1f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:33.665046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff1f6ff-7fd7-46e5-8362-0915bf8ced1f service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:33.674909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 941/1866] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:34.147756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e9fc3b-7be4-40cc-857c-e0a32995ec2d None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:34.149689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-82e9fc3b-7be4-40cc-857c-e0a32995ec2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:34.163670 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e9fc3b-7be4-40cc-857c-e0a32995ec2d None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:34.170316 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:34.171349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:34.171544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:34.171894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a admin admin] PATH_INFO: `/v3/projects/927bffce9c494f559693b747ea737c74` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:34.173721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a admin admin] RBAC: Authorizing `identity:delete_project(project_id=927bffce9c494f559693b747ea737c74)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:34.175427 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:34.248741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '927bffce9c494f559693b747ea737c74', 'request_id': 'req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:34.252914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:34.255417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '927bffce9c494f559693b747ea737c74', 'request_id': 'req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:34.264302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '927bffce9c494f559693b747ea737c74', 'request_id': 'req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:34.275558 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a admin admin] The token cache is being invalidate because project 927bffce9c494f559693b747ea737c74 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:34.275836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:34.276572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1c2f13e9-abeb-4e5f-b29d-00e77dce054a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:34.278197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 926/1867] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:34 2022] DELETE /identity/v3/projects/927bffce9c494f559693b747ea737c74 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:34.283687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28334470-8f0b-4a2d-bd9d-59824a2d92a1 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:34.284182 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28334470-8f0b-4a2d-bd9d-59824a2d92a1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:34.284389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28334470-8f0b-4a2d-bd9d-59824a2d92a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:34.305751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-28334470-8f0b-4a2d-bd9d-59824a2d92a1 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:34.306947 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-28334470-8f0b-4a2d-bd9d-59824a2d92a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:34.332468 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:34.386774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 942/1868] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:34.394442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6545a8-6399-4fee-86ed-2ec1e2551072 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:34.394838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6545a8-6399-4fee-86ed-2ec1e2551072 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:34.395120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6545a8-6399-4fee-86ed-2ec1e2551072 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:34.415740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-1a6545a8-6399-4fee-86ed-2ec1e2551072 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:34.416993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1a6545a8-6399-4fee-86ed-2ec1e2551072 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:34.426101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 927/1869] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:34.430526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28334470-8f0b-4a2d-bd9d-59824a2d92a1 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:34.440751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28334470-8f0b-4a2d-bd9d-59824a2d92a1 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:34.445971 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2b53c4-ec3b-42de-b1c1-d9e4d2a44657 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:34.446584 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2b53c4-ec3b-42de-b1c1-d9e4d2a44657 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:34.446908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2b53c4-ec3b-42de-b1c1-d9e4d2a44657 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:34.447191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2b53c4-ec3b-42de-b1c1-d9e4d2a44657 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:34.447983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2b53c4-ec3b-42de-b1c1-d9e4d2a44657 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:15:34.448287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2b53c4-ec3b-42de-b1c1-d9e4d2a44657 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:34.449944 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2b53c4-ec3b-42de-b1c1-d9e4d2a44657 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:34.462225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 943/1870] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:15:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:15:34.467088 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6545a8-6399-4fee-86ed-2ec1e2551072 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:34.478683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6545a8-6399-4fee-86ed-2ec1e2551072 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:34.486630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3dc0f3-0a81-43d1-8b99-db20caadfc39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:34.487259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3dc0f3-0a81-43d1-8b99-db20caadfc39 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:34.487540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3dc0f3-0a81-43d1-8b99-db20caadfc39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:34.487837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3dc0f3-0a81-43d1-8b99-db20caadfc39 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:34.488509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3dc0f3-0a81-43d1-8b99-db20caadfc39 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:34.490974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3dc0f3-0a81-43d1-8b99-db20caadfc39 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:34.523798 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 928/1871] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:34 2022] POST /identity/v3/projects => generated 375 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:34.524854 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2b53c4-ec3b-42de-b1c1-d9e4d2a44657 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:34.535213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2b53c4-ec3b-42de-b1c1-d9e4d2a44657 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:34.593631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdab8e2a-de63-4234-807d-fe34c55ac903 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:34.594491 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdab8e2a-de63-4234-807d-fe34c55ac903 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:34.594772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdab8e2a-de63-4234-807d-fe34c55ac903 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:34.595043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdab8e2a-de63-4234-807d-fe34c55ac903 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:34.595679 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdab8e2a-de63-4234-807d-fe34c55ac903 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:34.597700 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdab8e2a-de63-4234-807d-fe34c55ac903 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:34.618740 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 944/1872] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:34 2022] POST /identity/v3/users => generated 288 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:15:34.622961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3dc0f3-0a81-43d1-8b99-db20caadfc39 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:34.633288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3dc0f3-0a81-43d1-8b99-db20caadfc39 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:34.637864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f169bbb3-4c95-4069-9090-c0b094499bda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:34.638949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f169bbb3-4c95-4069-9090-c0b094499bda admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:34.639231 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f169bbb3-4c95-4069-9090-c0b094499bda admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:34.639504 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f169bbb3-4c95-4069-9090-c0b094499bda admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:34.640144 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f169bbb3-4c95-4069-9090-c0b094499bda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:34.641821 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f169bbb3-4c95-4069-9090-c0b094499bda admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:34.651582 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 929/1873] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:34 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:34.656376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdab8e2a-de63-4234-807d-fe34c55ac903 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:34.666123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdab8e2a-de63-4234-807d-fe34c55ac903 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:34.671313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b196609-c7f4-4da1-a410-e50bc5b641cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:34.671950 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b196609-c7f4-4da1-a410-e50bc5b641cc admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:34.672228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b196609-c7f4-4da1-a410-e50bc5b641cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:34.672527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b196609-c7f4-4da1-a410-e50bc5b641cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:34.673174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b196609-c7f4-4da1-a410-e50bc5b641cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:34.674817 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b196609-c7f4-4da1-a410-e50bc5b641cc admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:34.685476 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 945/1874] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:34 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:34.689711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f169bbb3-4c95-4069-9090-c0b094499bda admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:34.701186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f169bbb3-4c95-4069-9090-c0b094499bda admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:34.705943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b927e95-afa1-4256-8bf8-064bb57652cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:34.706620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b927e95-afa1-4256-8bf8-064bb57652cd admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:34.706899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b927e95-afa1-4256-8bf8-064bb57652cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:34.707173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b927e95-afa1-4256-8bf8-064bb57652cd admin admin] PATH_INFO: `/v3/projects/511c922df9194d8b95b86656107f1c99/users/334b1c2600404cd2a3cfaa97beab51c8/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:34.722331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b927e95-afa1-4256-8bf8-064bb57652cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=511c922df9194d8b95b86656107f1c99, user_id=334b1c2600404cd2a3cfaa97beab51c8, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:34.725030 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b927e95-afa1-4256-8bf8-064bb57652cd admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:34.737213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 930/1875] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:34 2022] PUT /identity/v3/projects/511c922df9194d8b95b86656107f1c99/users/334b1c2600404cd2a3cfaa97beab51c8/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:15:34.747549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd20d1e-aa5a-4581-b240-46bdbf0b4e4b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:34.747919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd20d1e-aa5a-4581-b240-46bdbf0b4e4b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:34.748196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd20d1e-aa5a-4581-b240-46bdbf0b4e4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:34.775422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-5dd20d1e-aa5a-4581-b240-46bdbf0b4e4b None None] MFA Rules not processed for user `334b1c2600404cd2a3cfaa97beab51c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:34.776631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5dd20d1e-aa5a-4581-b240-46bdbf0b4e4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:34.838907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 946/1876] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:34 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:34.850334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b927e95-afa1-4256-8bf8-064bb57652cd admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:34.852012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9b927e95-afa1-4256-8bf8-064bb57652cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:34.907590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b927e95-afa1-4256-8bf8-064bb57652cd admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:34.911206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa2565d-171f-47be-99f2-72b53327dccb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:34.911816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa2565d-171f-47be-99f2-72b53327dccb service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:34.912144 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa2565d-171f-47be-99f2-72b53327dccb service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:34.912653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa2565d-171f-47be-99f2-72b53327dccb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:34.926170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa2565d-171f-47be-99f2-72b53327dccb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:34.927627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa2565d-171f-47be-99f2-72b53327dccb service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:34.937221 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 931/1877] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:34 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:37.943220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd20d1e-aa5a-4581-b240-46bdbf0b4e4b None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:37.944852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5dd20d1e-aa5a-4581-b240-46bdbf0b4e4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:37.957181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd20d1e-aa5a-4581-b240-46bdbf0b4e4b None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:37.961334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74999fc0-e35b-4e84-b087-ffa1ae02e90e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:37.961935 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74999fc0-e35b-4e84-b087-ffa1ae02e90e service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:37.962214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74999fc0-e35b-4e84-b087-ffa1ae02e90e service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:37.962487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74999fc0-e35b-4e84-b087-ffa1ae02e90e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:37.964292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-74999fc0-e35b-4e84-b087-ffa1ae02e90e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:37.992969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:38.033596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74999fc0-e35b-4e84-b087-ffa1ae02e90e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:38.035156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74999fc0-e35b-4e84-b087-ffa1ae02e90e service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:38.045691 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 947/1878] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:37 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:38.644397 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a48958e-bbb5-4eb6-b313-fc9aa0d21f51 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:38.644836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a48958e-bbb5-4eb6-b313-fc9aa0d21f51 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:38.644836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a48958e-bbb5-4eb6-b313-fc9aa0d21f51 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:38.646311 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 932/1879] 10.222.0.26 () {54 vars in 921 bytes} [Wed Sep 28 12:15:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:15:38.654329 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6ec5e3-a228-400e-bb4c-d2866f9d0aff None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:38.654809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6ec5e3-a228-400e-bb4c-d2866f9d0aff None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:38.655195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6ec5e3-a228-400e-bb4c-d2866f9d0aff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:38.707847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-db6ec5e3-a228-400e-bb4c-d2866f9d0aff None None] MFA Rules not processed for user `8bf969348e7a4158acf8fb789430b0e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:38.711686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-db6ec5e3-a228-400e-bb4c-d2866f9d0aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:38.848000 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 948/1880] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Sep 28 12:15:38 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:38.909278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a48958e-bbb5-4eb6-b313-fc9aa0d21f51 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:38.912363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2a48958e-bbb5-4eb6-b313-fc9aa0d21f51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:38.931709 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a48958e-bbb5-4eb6-b313-fc9aa0d21f51 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:38.947338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961ac8bf-f58a-4fe9-8801-e4f966a3229f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:38.948127 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961ac8bf-f58a-4fe9-8801-e4f966a3229f service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:38.948503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961ac8bf-f58a-4fe9-8801-e4f966a3229f service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:38.948839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961ac8bf-f58a-4fe9-8801-e4f966a3229f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:38.959116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961ac8bf-f58a-4fe9-8801-e4f966a3229f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:38.961141 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961ac8bf-f58a-4fe9-8801-e4f966a3229f service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:38.972505 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 933/1881] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:15:40.196585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6ec5e3-a228-400e-bb4c-d2866f9d0aff None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:40.260300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6ec5e3-a228-400e-bb4c-d2866f9d0aff None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:40.265953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08f3de7-94d9-4ae9-bc92-5a4e9213530e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:40.266838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08f3de7-94d9-4ae9-bc92-5a4e9213530e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:40.267215 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08f3de7-94d9-4ae9-bc92-5a4e9213530e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:40.267602 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08f3de7-94d9-4ae9-bc92-5a4e9213530e admin admin] PATH_INFO: `/v3/users/334b1c2600404cd2a3cfaa97beab51c8` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:40.268866 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08f3de7-94d9-4ae9-bc92-5a4e9213530e admin admin] RBAC: Authorizing `identity:delete_user(user_id=334b1c2600404cd2a3cfaa97beab51c8)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:40.270767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08f3de7-94d9-4ae9-bc92-5a4e9213530e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:40.320489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c08f3de7-94d9-4ae9-bc92-5a4e9213530e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '334b1c2600404cd2a3cfaa97beab51c8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:40.321186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c08f3de7-94d9-4ae9-bc92-5a4e9213530e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '334b1c2600404cd2a3cfaa97beab51c8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:40.321342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961ac8bf-f58a-4fe9-8801-e4f966a3229f service nova] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:40.322945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-961ac8bf-f58a-4fe9-8801-e4f966a3229f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:40.326793 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:40.329026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c08f3de7-94d9-4ae9-bc92-5a4e9213530e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '334b1c2600404cd2a3cfaa97beab51c8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:40.329738 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c08f3de7-94d9-4ae9-bc92-5a4e9213530e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '334b1c2600404cd2a3cfaa97beab51c8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:40.336614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961ac8bf-f58a-4fe9-8801-e4f966a3229f service nova] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:40.341660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537a4575-d8fe-4930-9bdd-d8c3edd262c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:40.342270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a4575-d8fe-4930-9bdd-d8c3edd262c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:40.342544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a4575-d8fe-4930-9bdd-d8c3edd262c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:40.342865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a4575-d8fe-4930-9bdd-d8c3edd262c9 admin admin] PATH_INFO: `/v3/users/3b24c0835d8d4d2aa39940b9414a3da1` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:40.344439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537a4575-d8fe-4930-9bdd-d8c3edd262c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3b24c0835d8d4d2aa39940b9414a3da1)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:40.344818 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c08f3de7-94d9-4ae9-bc92-5a4e9213530e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '334b1c2600404cd2a3cfaa97beab51c8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:40.346342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537a4575-d8fe-4930-9bdd-d8c3edd262c9 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:40.358784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 949/1882] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:40 2022] DELETE /identity/v3/users/334b1c2600404cd2a3cfaa97beab51c8 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:40.363086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c6da6f-1ab2-4253-ad59-ea5127236675 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:40.363371 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c6da6f-1ab2-4253-ad59-ea5127236675 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:40.363652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c6da6f-1ab2-4253-ad59-ea5127236675 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:40.385708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-23c6da6f-1ab2-4253-ad59-ea5127236675 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:40.386928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-23c6da6f-1ab2-4253-ad59-ea5127236675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:40.398567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-537a4575-d8fe-4930-9bdd-d8c3edd262c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3b24c0835d8d4d2aa39940b9414a3da1'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:40.399219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-537a4575-d8fe-4930-9bdd-d8c3edd262c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3b24c0835d8d4d2aa39940b9414a3da1'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:40.399929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-537a4575-d8fe-4930-9bdd-d8c3edd262c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3b24c0835d8d4d2aa39940b9414a3da1'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:40.407474 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-537a4575-d8fe-4930-9bdd-d8c3edd262c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3b24c0835d8d4d2aa39940b9414a3da1'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:40.412961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:DELETE=10,keystone:SELECT=87,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:40.422017 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-537a4575-d8fe-4930-9bdd-d8c3edd262c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3b24c0835d8d4d2aa39940b9414a3da1'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:40.436331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 934/1883] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:40 2022] DELETE /identity/v3/users/3b24c0835d8d4d2aa39940b9414a3da1 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:15:40.440568 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5c97dc-7e8b-43a9-ab19-0add866857f1 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:40.441164 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5c97dc-7e8b-43a9-ab19-0add866857f1 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:40.441542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5c97dc-7e8b-43a9-ab19-0add866857f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:40.450304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 950/1884] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:40.460021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c6da6f-1ab2-4253-ad59-ea5127236675 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:40.460580 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-7e5c97dc-7e8b-43a9-ab19-0add866857f1 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:40.461388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-23c6da6f-1ab2-4253-ad59-ea5127236675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:40.461782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-7e5c97dc-7e8b-43a9-ab19-0add866857f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:40.523570 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c6da6f-1ab2-4253-ad59-ea5127236675 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:40.532921 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 935/1885] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:40.543740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5c97dc-7e8b-43a9-ab19-0add866857f1 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:40.546131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-7e5c97dc-7e8b-43a9-ab19-0add866857f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:40.577915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafa2c0f-68a0-413b-9b21-334317b291f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:40.578528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafa2c0f-68a0-413b-9b21-334317b291f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:40.578809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafa2c0f-68a0-413b-9b21-334317b291f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:40.579084 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafa2c0f-68a0-413b-9b21-334317b291f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:40.590420 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafa2c0f-68a0-413b-9b21-334317b291f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:40.592528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafa2c0f-68a0-413b-9b21-334317b291f6 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:40.605438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 951/1886] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:15:40.611253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5c97dc-7e8b-43a9-ab19-0add866857f1 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:40.615386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72107302-f600-4f59-98e6-c51728194886 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:40.616147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72107302-f600-4f59-98e6-c51728194886 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:40.616579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72107302-f600-4f59-98e6-c51728194886 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:40.617058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72107302-f600-4f59-98e6-c51728194886 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:40.627760 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72107302-f600-4f59-98e6-c51728194886 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:40.629241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72107302-f600-4f59-98e6-c51728194886 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:40.639734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 936/1887] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:15:40.990308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafa2c0f-68a0-413b-9b21-334317b291f6 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:40.991920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-eafa2c0f-68a0-413b-9b21-334317b291f6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:41.005993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafa2c0f-68a0-413b-9b21-334317b291f6 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:41.012677 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:41.013555 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:41.014130 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:41.014286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0 admin admin] PATH_INFO: `/v3/projects/511c922df9194d8b95b86656107f1c99` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:41.015465 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=511c922df9194d8b95b86656107f1c99)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:41.017990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:41.054191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:41.070157 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72107302-f600-4f59-98e6-c51728194886 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:41.072386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-72107302-f600-4f59-98e6-c51728194886 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:41.073354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '511c922df9194d8b95b86656107f1c99', 'request_id': 'req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.081170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '511c922df9194d8b95b86656107f1c99', 'request_id': 'req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.086562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72107302-f600-4f59-98e6-c51728194886 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:41.090556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '511c922df9194d8b95b86656107f1c99', 'request_id': 'req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.107632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0 admin admin] The token cache is being invalidate because project 511c922df9194d8b95b86656107f1c99 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:41.108102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.108881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8e0ef7d7-5d1e-47d0-90a6-6c0b82dfcec0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.110987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 952/1888] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:40 2022] DELETE /identity/v3/projects/511c922df9194d8b95b86656107f1c99 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:41.115328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a311f-ff0d-4ec1-aaf9-a6aee67a4e38 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:41.115611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a311f-ff0d-4ec1-aaf9-a6aee67a4e38 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:41.115887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a311f-ff0d-4ec1-aaf9-a6aee67a4e38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:41.135609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-ee8a311f-ff0d-4ec1-aaf9-a6aee67a4e38 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:41.136857 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ee8a311f-ff0d-4ec1-aaf9-a6aee67a4e38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:41.167136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:41.168910 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:41.169890 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:41.170239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] PATH_INFO: `/v3/projects/c9277e00edf8427c942ade41ca99e5bf` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:41.171466 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c9277e00edf8427c942ade41ca99e5bf)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:41.173567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:41.206516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 953/1889] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:41.214776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:41.216618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da50fe-9458-4821-a13b-4d06c015b03b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:41.216968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da50fe-9458-4821-a13b-4d06c015b03b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:41.217731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da50fe-9458-4821-a13b-4d06c015b03b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:41.240676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-d5da50fe-9458-4821-a13b-4d06c015b03b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:41.242553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d5da50fe-9458-4821-a13b-4d06c015b03b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:41.242816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c9277e00edf8427c942ade41ca99e5bf', 'request_id': 'req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.250899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c9277e00edf8427c942ade41ca99e5bf', 'request_id': 'req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.266066 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c9277e00edf8427c942ade41ca99e5bf', 'request_id': 'req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.287187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] The token cache is being invalidate because project c9277e00edf8427c942ade41ca99e5bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:41.287482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.290753 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.296333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 937/1890] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:41 2022] DELETE /identity/v3/projects/c9277e00edf8427c942ade41ca99e5bf => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:41.302530 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:41.305902 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:41.318098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 954/1891] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:41.342259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5da50fe-9458-4821-a13b-4d06c015b03b None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:41.360012 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5da50fe-9458-4821-a13b-4d06c015b03b None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:41.364628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c7f7f5-f461-4391-86ed-3828aeff3b96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:41.365240 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c7f7f5-f461-4391-86ed-3828aeff3b96 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:41.366083 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c7f7f5-f461-4391-86ed-3828aeff3b96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:41.366983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c7f7f5-f461-4391-86ed-3828aeff3b96 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:41.367676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c7f7f5-f461-4391-86ed-3828aeff3b96 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:15:41.368574 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c7f7f5-f461-4391-86ed-3828aeff3b96 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:41.370557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c7f7f5-f461-4391-86ed-3828aeff3b96 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:41.387663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 955/1892] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:15:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:15:41.389869 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c7f7f5-f461-4391-86ed-3828aeff3b96 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:41.396194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7382c2d2-f6f6-4e23-8faa-fe0a82cffc50 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:41.400157 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c7f7f5-f461-4391-86ed-3828aeff3b96 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:41.401885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a553e70b-3fc4-4849-887f-b181ac57e0df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:41.402622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a553e70b-3fc4-4849-887f-b181ac57e0df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:41.403018 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a553e70b-3fc4-4849-887f-b181ac57e0df admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:41.403573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a553e70b-3fc4-4849-887f-b181ac57e0df admin admin] PATH_INFO: `/v3/users/5da1a120b31f406894d206a959148ed0` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:41.405368 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a553e70b-3fc4-4849-887f-b181ac57e0df admin admin] RBAC: Authorizing `identity:delete_user(user_id=5da1a120b31f406894d206a959148ed0)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:41.409602 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4992561-d1ae-4ac4-843d-110027e06f29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:41.410196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4992561-d1ae-4ac4-843d-110027e06f29 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:41.410424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a553e70b-3fc4-4849-887f-b181ac57e0df admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:41.410536 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4992561-d1ae-4ac4-843d-110027e06f29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:41.410818 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4992561-d1ae-4ac4-843d-110027e06f29 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:41.411466 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4992561-d1ae-4ac4-843d-110027e06f29 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:41.413401 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4992561-d1ae-4ac4-843d-110027e06f29 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:41.448861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 956/1893] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:41 2022] POST /identity/v3/projects => generated 373 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:41.454029 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4992561-d1ae-4ac4-843d-110027e06f29 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:41.469563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a553e70b-3fc4-4849-887f-b181ac57e0df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5da1a120b31f406894d206a959148ed0'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.470623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a553e70b-3fc4-4849-887f-b181ac57e0df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5da1a120b31f406894d206a959148ed0'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.471742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a553e70b-3fc4-4849-887f-b181ac57e0df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5da1a120b31f406894d206a959148ed0'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.479403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:41.480856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a553e70b-3fc4-4849-887f-b181ac57e0df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5da1a120b31f406894d206a959148ed0'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.494123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a553e70b-3fc4-4849-887f-b181ac57e0df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5da1a120b31f406894d206a959148ed0'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.509032 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 938/1894] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:41 2022] DELETE /identity/v3/users/5da1a120b31f406894d206a959148ed0 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:41.539994 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4992561-d1ae-4ac4-843d-110027e06f29 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:41.602897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b566ebe-35fa-4972-a97e-2cd83457d7eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:41.603631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b566ebe-35fa-4972-a97e-2cd83457d7eb admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:41.603944 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b566ebe-35fa-4972-a97e-2cd83457d7eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:41.604195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b566ebe-35fa-4972-a97e-2cd83457d7eb admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:41.604875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b566ebe-35fa-4972-a97e-2cd83457d7eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:41.606849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b566ebe-35fa-4972-a97e-2cd83457d7eb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:41.627669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 957/1895] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:41 2022] POST /identity/v3/users => generated 287 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:15:41.631811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a553e70b-3fc4-4849-887f-b181ac57e0df admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:41.633569 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a553e70b-3fc4-4849-887f-b181ac57e0df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:41.647757 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a553e70b-3fc4-4849-887f-b181ac57e0df admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:41.652674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bf8e2a-9520-43eb-937c-aa3589367226 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:41.653295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bf8e2a-9520-43eb-937c-aa3589367226 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:41.654101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bf8e2a-9520-43eb-937c-aa3589367226 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:41.654377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bf8e2a-9520-43eb-937c-aa3589367226 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:41.655019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bf8e2a-9520-43eb-937c-aa3589367226 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:41.661660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bf8e2a-9520-43eb-937c-aa3589367226 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:41.676952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 939/1896] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:41 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:41.678339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b566ebe-35fa-4972-a97e-2cd83457d7eb admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:41.689389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b566ebe-35fa-4972-a97e-2cd83457d7eb admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:41.695379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c993c0a6-c6d7-4b68-b382-63c57d8e754d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:41.695979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c993c0a6-c6d7-4b68-b382-63c57d8e754d admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:41.696256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c993c0a6-c6d7-4b68-b382-63c57d8e754d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:41.696569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c993c0a6-c6d7-4b68-b382-63c57d8e754d admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:41.697263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c993c0a6-c6d7-4b68-b382-63c57d8e754d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:41.698762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c993c0a6-c6d7-4b68-b382-63c57d8e754d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:41.710756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 958/1897] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:41 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:41.715092 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bf8e2a-9520-43eb-937c-aa3589367226 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:41.726249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bf8e2a-9520-43eb-937c-aa3589367226 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:41.732003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3772acf1-1ba2-4684-9367-64150a44edb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:41.732885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3772acf1-1ba2-4684-9367-64150a44edb8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:41.733284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3772acf1-1ba2-4684-9367-64150a44edb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:41.733521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3772acf1-1ba2-4684-9367-64150a44edb8 admin admin] PATH_INFO: `/v3/projects/7cde07ab829342a5a57acc65f455a1af/users/eae40347e449436184b62c3dd33966f0/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:41.755300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3772acf1-1ba2-4684-9367-64150a44edb8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7cde07ab829342a5a57acc65f455a1af, user_id=eae40347e449436184b62c3dd33966f0, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:41.758254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3772acf1-1ba2-4684-9367-64150a44edb8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:41.766689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:DELETE=16,keystone:SELECT=81,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:41.773232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 940/1898] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:41 2022] PUT /identity/v3/projects/7cde07ab829342a5a57acc65f455a1af/users/eae40347e449436184b62c3dd33966f0/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:41.785277 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50567f-d915-4b13-88b8-fd34a5c6b233 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:41.785566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50567f-d915-4b13-88b8-fd34a5c6b233 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:41.785844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50567f-d915-4b13-88b8-fd34a5c6b233 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:41.802632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3772acf1-1ba2-4684-9367-64150a44edb8 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:41.804558 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3772acf1-1ba2-4684-9367-64150a44edb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:41.822001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4c50567f-d915-4b13-88b8-fd34a5c6b233 None None] MFA Rules not processed for user `eae40347e449436184b62c3dd33966f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:41.823564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4c50567f-d915-4b13-88b8-fd34a5c6b233 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:41.892193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3772acf1-1ba2-4684-9367-64150a44edb8 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:41.897295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ed01a8-4a66-4d89-9b71-127f0810b336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:41.898128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ed01a8-4a66-4d89-9b71-127f0810b336 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:41.898649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ed01a8-4a66-4d89-9b71-127f0810b336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:41.898997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ed01a8-4a66-4d89-9b71-127f0810b336 admin admin] PATH_INFO: `/v3/projects/dfa48c05bb544e3495431dc2d6c016d0` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:41.900810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ed01a8-4a66-4d89-9b71-127f0810b336 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dfa48c05bb544e3495431dc2d6c016d0)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:41.902804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ed01a8-4a66-4d89-9b71-127f0810b336 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:41.910096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 959/1899] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:41.922708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c50567f-d915-4b13-88b8-fd34a5c6b233 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:41.925469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4c50567f-d915-4b13-88b8-fd34a5c6b233 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:41.975803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:41.984728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-b8ed01a8-4a66-4d89-9b71-127f0810b336 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dfa48c05bb544e3495431dc2d6c016d0', 'request_id': 'req-b8ed01a8-4a66-4d89-9b71-127f0810b336'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:41.996028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-b8ed01a8-4a66-4d89-9b71-127f0810b336 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dfa48c05bb544e3495431dc2d6c016d0', 'request_id': 'req-b8ed01a8-4a66-4d89-9b71-127f0810b336'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:42.006657 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c50567f-d915-4b13-88b8-fd34a5c6b233 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:42.010399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-b8ed01a8-4a66-4d89-9b71-127f0810b336 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dfa48c05bb544e3495431dc2d6c016d0', 'request_id': 'req-b8ed01a8-4a66-4d89-9b71-127f0810b336'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:42.033927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-b8ed01a8-4a66-4d89-9b71-127f0810b336 admin admin] The token cache is being invalidate because project dfa48c05bb544e3495431dc2d6c016d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:42.034887 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-b8ed01a8-4a66-4d89-9b71-127f0810b336 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:42.035228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-b8ed01a8-4a66-4d89-9b71-127f0810b336 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:42.037121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 941/1900] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:41 2022] DELETE /identity/v3/projects/dfa48c05bb544e3495431dc2d6c016d0 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:42.042083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75f7d09-5769-480b-a8f3-e80f1d9b77bd None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:42.043019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75f7d09-5769-480b-a8f3-e80f1d9b77bd None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:42.043019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75f7d09-5769-480b-a8f3-e80f1d9b77bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:42.076346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-e75f7d09-5769-480b-a8f3-e80f1d9b77bd None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:42.076346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e75f7d09-5769-480b-a8f3-e80f1d9b77bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:42.082971 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29d9011-8e4e-418d-9cc5-777db7c55354 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:42.083754 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29d9011-8e4e-418d-9cc5-777db7c55354 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:42.084058 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29d9011-8e4e-418d-9cc5-777db7c55354 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:42.084354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29d9011-8e4e-418d-9cc5-777db7c55354 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:42.086538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d29d9011-8e4e-418d-9cc5-777db7c55354 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:42.135315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 942/1901] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:42.144238 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceb845b-c075-4d4a-bdb9-0197d9c1f735 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:42.144582 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceb845b-c075-4d4a-bdb9-0197d9c1f735 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:42.144867 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceb845b-c075-4d4a-bdb9-0197d9c1f735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:42.150244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29d9011-8e4e-418d-9cc5-777db7c55354 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:42.151763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29d9011-8e4e-418d-9cc5-777db7c55354 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:42.164109 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 960/1902] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:41 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:42.169805 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3ceb845b-c075-4d4a-bdb9-0197d9c1f735 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:42.171074 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3ceb845b-c075-4d4a-bdb9-0197d9c1f735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:42.180354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 943/1903] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:42.184722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29d9011-8e4e-418d-9cc5-777db7c55354 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:42.228217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29d9011-8e4e-418d-9cc5-777db7c55354 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:42.239784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc925320-b2ee-443b-ab5e-b0f0327eec83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:42.240939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc925320-b2ee-443b-ab5e-b0f0327eec83 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:42.241264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc925320-b2ee-443b-ab5e-b0f0327eec83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:42.242000 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc925320-b2ee-443b-ab5e-b0f0327eec83 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:42.242677 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc925320-b2ee-443b-ab5e-b0f0327eec83 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:15:42.242996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc925320-b2ee-443b-ab5e-b0f0327eec83 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:42.245075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc925320-b2ee-443b-ab5e-b0f0327eec83 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:42.265007 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 961/1904] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:15:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:15:42.266123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ceb845b-c075-4d4a-bdb9-0197d9c1f735 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:42.279553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ceb845b-c075-4d4a-bdb9-0197d9c1f735 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:42.285086 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415dd3fb-34d3-44ea-9f3c-eaf2f8c43b4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:42.285692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415dd3fb-34d3-44ea-9f3c-eaf2f8c43b4c admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:42.285978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415dd3fb-34d3-44ea-9f3c-eaf2f8c43b4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:42.286282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415dd3fb-34d3-44ea-9f3c-eaf2f8c43b4c admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:42.286995 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415dd3fb-34d3-44ea-9f3c-eaf2f8c43b4c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:42.289000 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415dd3fb-34d3-44ea-9f3c-eaf2f8c43b4c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:42.316355 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:42.333469 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 944/1905] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:42 2022] POST /identity/v3/projects => generated 391 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:42.334983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc925320-b2ee-443b-ab5e-b0f0327eec83 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:42.351088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc925320-b2ee-443b-ab5e-b0f0327eec83 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:42.457088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec9480e-4f86-4be8-b3e9-1a5f26f228b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:42.458611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec9480e-4f86-4be8-b3e9-1a5f26f228b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:42.460090 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec9480e-4f86-4be8-b3e9-1a5f26f228b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:42.461757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec9480e-4f86-4be8-b3e9-1a5f26f228b9 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:42.464555 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec9480e-4f86-4be8-b3e9-1a5f26f228b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:42.470540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec9480e-4f86-4be8-b3e9-1a5f26f228b9 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:42.544148 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 962/1906] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:42 2022] POST /identity/v3/users => generated 296 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:42.545919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415dd3fb-34d3-44ea-9f3c-eaf2f8c43b4c admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:42.561710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415dd3fb-34d3-44ea-9f3c-eaf2f8c43b4c admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:42.569203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8131dc4-3fc6-43d1-a23a-9c945f28c509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:42.570058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8131dc4-3fc6-43d1-a23a-9c945f28c509 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:42.570278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8131dc4-3fc6-43d1-a23a-9c945f28c509 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:42.570637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8131dc4-3fc6-43d1-a23a-9c945f28c509 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:42.571320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8131dc4-3fc6-43d1-a23a-9c945f28c509 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:42.573283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8131dc4-3fc6-43d1-a23a-9c945f28c509 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:42.589142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 945/1907] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:42 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:42.590514 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec9480e-4f86-4be8-b3e9-1a5f26f228b9 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:42.603662 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec9480e-4f86-4be8-b3e9-1a5f26f228b9 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:42.610625 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04659bf4-66c5-4de1-98a9-7865ba695c0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:42.611658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04659bf4-66c5-4de1-98a9-7865ba695c0b admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:42.612691 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04659bf4-66c5-4de1-98a9-7865ba695c0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:42.612691 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04659bf4-66c5-4de1-98a9-7865ba695c0b admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:42.612955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04659bf4-66c5-4de1-98a9-7865ba695c0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:42.614963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04659bf4-66c5-4de1-98a9-7865ba695c0b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:42.635517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 963/1908] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:42 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:15:42.636502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8131dc4-3fc6-43d1-a23a-9c945f28c509 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:42.656478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8131dc4-3fc6-43d1-a23a-9c945f28c509 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:42.663519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b23e9d-4c7c-49d2-8290-3fc2dcc11f54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:42.664273 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b23e9d-4c7c-49d2-8290-3fc2dcc11f54 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:42.664563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b23e9d-4c7c-49d2-8290-3fc2dcc11f54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:42.664845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b23e9d-4c7c-49d2-8290-3fc2dcc11f54 admin admin] PATH_INFO: `/v3/projects/7e9f82c2dd784c2a837ec44457201aad/users/cdadc0b0cde44cfabe4cba6103a59842/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:42.702480 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b23e9d-4c7c-49d2-8290-3fc2dcc11f54 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e9f82c2dd784c2a837ec44457201aad, user_id=cdadc0b0cde44cfabe4cba6103a59842, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:42.705402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b23e9d-4c7c-49d2-8290-3fc2dcc11f54 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:42.725832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 946/1909] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:42 2022] PUT /identity/v3/projects/7e9f82c2dd784c2a837ec44457201aad/users/cdadc0b0cde44cfabe4cba6103a59842/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:42.737709 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b1ee53-dfd5-47f8-83cd-54272a4062c0 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:42.737997 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b1ee53-dfd5-47f8-83cd-54272a4062c0 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:42.738262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b1ee53-dfd5-47f8-83cd-54272a4062c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:42.784454 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:42.821918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-c9b1ee53-dfd5-47f8-83cd-54272a4062c0 None None] MFA Rules not processed for user `cdadc0b0cde44cfabe4cba6103a59842`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:42.823588 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c9b1ee53-dfd5-47f8-83cd-54272a4062c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:42.916792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 964/1910] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:42 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:42.921574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b23e9d-4c7c-49d2-8290-3fc2dcc11f54 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:43.002186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b23e9d-4c7c-49d2-8290-3fc2dcc11f54 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:43.007320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edeea66-2e92-49bd-bca1-321e27829be2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:43.008056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edeea66-2e92-49bd-bca1-321e27829be2 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:43.008472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edeea66-2e92-49bd-bca1-321e27829be2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:43.008820 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edeea66-2e92-49bd-bca1-321e27829be2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:43.009527 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edeea66-2e92-49bd-bca1-321e27829be2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:43.011634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edeea66-2e92-49bd-bca1-321e27829be2 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:43.042270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 947/1911] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:42 2022] POST /identity/v3/projects => generated 393 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:43.046309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b1ee53-dfd5-47f8-83cd-54272a4062c0 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:43.056986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b1ee53-dfd5-47f8-83cd-54272a4062c0 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:43.103189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edeea66-2e92-49bd-bca1-321e27829be2 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:43.105330 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1edeea66-2e92-49bd-bca1-321e27829be2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:43.129265 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b8ad7e-8139-44fa-b31d-26b9ee443eea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:43.129967 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b8ad7e-8139-44fa-b31d-26b9ee443eea admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:43.130315 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b8ad7e-8139-44fa-b31d-26b9ee443eea admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:43.130652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b8ad7e-8139-44fa-b31d-26b9ee443eea admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:43.131410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b8ad7e-8139-44fa-b31d-26b9ee443eea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:43.133550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b8ad7e-8139-44fa-b31d-26b9ee443eea admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:43.157932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 965/1912] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:43 2022] POST /identity/v3/users => generated 303 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:43.162423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b8ad7e-8139-44fa-b31d-26b9ee443eea admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:43.174699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b8ad7e-8139-44fa-b31d-26b9ee443eea admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:43.179979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94b5b49-2734-40c3-abcd-4d03ba5a1656 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:43.180736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94b5b49-2734-40c3-abcd-4d03ba5a1656 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:43.181120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94b5b49-2734-40c3-abcd-4d03ba5a1656 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:43.181497 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94b5b49-2734-40c3-abcd-4d03ba5a1656 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:43.182266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94b5b49-2734-40c3-abcd-4d03ba5a1656 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:43.183943 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94b5b49-2734-40c3-abcd-4d03ba5a1656 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:43.189771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edeea66-2e92-49bd-bca1-321e27829be2 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:43.194894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 966/1913] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:43 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:15:43.195027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ce413e-db25-4c2b-a80a-01829badda7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:43.195700 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ce413e-db25-4c2b-a80a-01829badda7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:43.196093 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ce413e-db25-4c2b-a80a-01829badda7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:43.196476 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ce413e-db25-4c2b-a80a-01829badda7c admin admin] PATH_INFO: `/v3/users/7fe58b0d4f6646ce9209b5d323476076` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:43.197689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ce413e-db25-4c2b-a80a-01829badda7c admin admin] RBAC: Authorizing `identity:delete_user(user_id=7fe58b0d4f6646ce9209b5d323476076)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:43.199488 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94b5b49-2734-40c3-abcd-4d03ba5a1656 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:43.199942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ce413e-db25-4c2b-a80a-01829badda7c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:43.213527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94b5b49-2734-40c3-abcd-4d03ba5a1656 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:43.220163 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026dcd7f-d579-4244-8877-75f7bf595f94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:43.220865 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026dcd7f-d579-4244-8877-75f7bf595f94 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:43.221147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026dcd7f-d579-4244-8877-75f7bf595f94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:43.221457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026dcd7f-d579-4244-8877-75f7bf595f94 admin admin] PATH_INFO: `/v3/projects/8c78391efde441cd81b1fd99a56e45c9/users/d86be4405e5842558bef13938bff29c8/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:43.241835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026dcd7f-d579-4244-8877-75f7bf595f94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8c78391efde441cd81b1fd99a56e45c9, user_id=d86be4405e5842558bef13938bff29c8, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:43.248566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026dcd7f-d579-4244-8877-75f7bf595f94 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:43.256696 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e4ce413e-db25-4c2b-a80a-01829badda7c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7fe58b0d4f6646ce9209b5d323476076'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:43.257541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e4ce413e-db25-4c2b-a80a-01829badda7c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7fe58b0d4f6646ce9209b5d323476076'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:43.258199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e4ce413e-db25-4c2b-a80a-01829badda7c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7fe58b0d4f6646ce9209b5d323476076'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:43.263958 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:43.268651 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 967/1914] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:43 2022] PUT /identity/v3/projects/8c78391efde441cd81b1fd99a56e45c9/users/d86be4405e5842558bef13938bff29c8/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:15:43.273023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e4ce413e-db25-4c2b-a80a-01829badda7c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7fe58b0d4f6646ce9209b5d323476076'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:43.273375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026dcd7f-d579-4244-8877-75f7bf595f94 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:43.275004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-026dcd7f-d579-4244-8877-75f7bf595f94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:43.285243 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e4ce413e-db25-4c2b-a80a-01829badda7c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7fe58b0d4f6646ce9209b5d323476076'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:43.299206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 948/1915] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:43 2022] DELETE /identity/v3/users/7fe58b0d4f6646ce9209b5d323476076 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:15:43.303673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b5571d-863d-497e-8d55-bc4cabbf7e49 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:43.303957 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b5571d-863d-497e-8d55-bc4cabbf7e49 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:43.304256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b5571d-863d-497e-8d55-bc4cabbf7e49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:43.323478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-50b5571d-863d-497e-8d55-bc4cabbf7e49 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:43.324715 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-50b5571d-863d-497e-8d55-bc4cabbf7e49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:43.340565 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:43.346403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026dcd7f-d579-4244-8877-75f7bf595f94 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:43.385564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 949/1916] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:43.396447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b5571d-863d-497e-8d55-bc4cabbf7e49 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:43.397823 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-50b5571d-863d-497e-8d55-bc4cabbf7e49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:43.407851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9797be7c-f87d-42a0-a95b-72dd2b2403ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:43.408551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9797be7c-f87d-42a0-a95b-72dd2b2403ff admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:43.408781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9797be7c-f87d-42a0-a95b-72dd2b2403ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:43.409054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9797be7c-f87d-42a0-a95b-72dd2b2403ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:43.409696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9797be7c-f87d-42a0-a95b-72dd2b2403ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:43.411330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9797be7c-f87d-42a0-a95b-72dd2b2403ff admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:43.421173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 968/1917] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:43 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:15:43.425240 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9797be7c-f87d-42a0-a95b-72dd2b2403ff admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:43.434454 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9797be7c-f87d-42a0-a95b-72dd2b2403ff admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:43.439743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084e7b2c-a009-4521-8f6e-0b7542c2b5bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:43.440391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084e7b2c-a009-4521-8f6e-0b7542c2b5bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:43.440669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084e7b2c-a009-4521-8f6e-0b7542c2b5bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:43.440939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084e7b2c-a009-4521-8f6e-0b7542c2b5bd admin admin] PATH_INFO: `/v3/projects/8c78391efde441cd81b1fd99a56e45c9/users/d86be4405e5842558bef13938bff29c8/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:43.442793 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084e7b2c-a009-4521-8f6e-0b7542c2b5bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=8c78391efde441cd81b1fd99a56e45c9, user_id=d86be4405e5842558bef13938bff29c8, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:43.445430 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084e7b2c-a009-4521-8f6e-0b7542c2b5bd admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:43.455954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 969/1918] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:43 2022] PUT /identity/v3/projects/8c78391efde441cd81b1fd99a56e45c9/users/d86be4405e5842558bef13938bff29c8/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:15:43.466215 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fca6e20-349f-4d76-b10b-ec5100b9ede6 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:43.466492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fca6e20-349f-4d76-b10b-ec5100b9ede6 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:43.466762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fca6e20-349f-4d76-b10b-ec5100b9ede6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:43.467636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b5571d-863d-497e-8d55-bc4cabbf7e49 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:43.499123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-8fca6e20-349f-4d76-b10b-ec5100b9ede6 None None] MFA Rules not processed for user `d86be4405e5842558bef13938bff29c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:43.500607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8fca6e20-349f-4d76-b10b-ec5100b9ede6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:43.533690 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b7a0c7-853b-43ed-aae5-0c488ae5ce13 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:43.534300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b7a0c7-853b-43ed-aae5-0c488ae5ce13 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:43.534575 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b7a0c7-853b-43ed-aae5-0c488ae5ce13 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:43.534842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b7a0c7-853b-43ed-aae5-0c488ae5ce13 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:43.579532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 970/1919] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:43 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:43.588841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:43.595909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fca6e20-349f-4d76-b10b-ec5100b9ede6 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:43.598034 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8fca6e20-349f-4d76-b10b-ec5100b9ede6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:43.614373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b7a0c7-853b-43ed-aae5-0c488ae5ce13 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:43.616176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b7a0c7-853b-43ed-aae5-0c488ae5ce13 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:43.627343 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 950/1920] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:15:43.660114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fca6e20-349f-4d76-b10b-ec5100b9ede6 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:43.663894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349f17a1-fbd3-4fd3-a3b4-9df826356ef8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:43.664623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349f17a1-fbd3-4fd3-a3b4-9df826356ef8 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:43.664907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349f17a1-fbd3-4fd3-a3b4-9df826356ef8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:43.665218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349f17a1-fbd3-4fd3-a3b4-9df826356ef8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:43.677896 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349f17a1-fbd3-4fd3-a3b4-9df826356ef8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:43.680148 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349f17a1-fbd3-4fd3-a3b4-9df826356ef8 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:43.692577 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 971/1921] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:43 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:15:43.751646 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b7a0c7-853b-43ed-aae5-0c488ae5ce13 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:43.762537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b7a0c7-853b-43ed-aae5-0c488ae5ce13 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:43.767516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbeaa70-241a-45d5-a8d8-86e3596346ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:43.768239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbeaa70-241a-45d5-a8d8-86e3596346ae service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:43.768763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbeaa70-241a-45d5-a8d8-86e3596346ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:43.769101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbeaa70-241a-45d5-a8d8-86e3596346ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:43.770943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2fbeaa70-241a-45d5-a8d8-86e3596346ae service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:43.910929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbeaa70-241a-45d5-a8d8-86e3596346ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:43.913509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbeaa70-241a-45d5-a8d8-86e3596346ae service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:43.930676 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 951/1922] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:43 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:44.073867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349f17a1-fbd3-4fd3-a3b4-9df826356ef8 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:44.076645 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-349f17a1-fbd3-4fd3-a3b4-9df826356ef8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:44.115617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349f17a1-fbd3-4fd3-a3b4-9df826356ef8 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:44.128264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:44.128930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:44.129251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:44.129473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] PATH_INFO: `/v3/projects/c3e76fef546d4152bc0d60dfc4dc4304` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:44.130869 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c3e76fef546d4152bc0d60dfc4dc4304)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:44.132759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:44.152546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:44.255888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c3e76fef546d4152bc0d60dfc4dc4304', 'request_id': 'req-dbd6e030-266c-4553-94bd-63796387a863'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:44.266866 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c3e76fef546d4152bc0d60dfc4dc4304', 'request_id': 'req-dbd6e030-266c-4553-94bd-63796387a863'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:44.280425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c3e76fef546d4152bc0d60dfc4dc4304', 'request_id': 'req-dbd6e030-266c-4553-94bd-63796387a863'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:44.298501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] The token cache is being invalidate because project c3e76fef546d4152bc0d60dfc4dc4304 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:44.298827 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:44.302362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:44.321062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 972/1923] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:44 2022] DELETE /identity/v3/projects/c3e76fef546d4152bc0d60dfc4dc4304 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:44.323624 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbeaa70-241a-45d5-a8d8-86e3596346ae service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:44.342154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbeaa70-241a-45d5-a8d8-86e3596346ae service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:44.445914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a4256f-a812-4961-8aff-ca753045e02c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:44.446632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a4256f-a812-4961-8aff-ca753045e02c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:44.446981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a4256f-a812-4961-8aff-ca753045e02c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:44.447595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a4256f-a812-4961-8aff-ca753045e02c admin admin] PATH_INFO: `/v3/users/b87f7c8626eb4ec2838eb721ecfc6289` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:44.449239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a4256f-a812-4961-8aff-ca753045e02c admin admin] RBAC: Authorizing `identity:delete_user(user_id=b87f7c8626eb4ec2838eb721ecfc6289)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:44.451829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a4256f-a812-4961-8aff-ca753045e02c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:44.522529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-16a4256f-a812-4961-8aff-ca753045e02c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b87f7c8626eb4ec2838eb721ecfc6289'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:44.524195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-16a4256f-a812-4961-8aff-ca753045e02c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b87f7c8626eb4ec2838eb721ecfc6289'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:44.526127 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-16a4256f-a812-4961-8aff-ca753045e02c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b87f7c8626eb4ec2838eb721ecfc6289'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:44.535256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-16a4256f-a812-4961-8aff-ca753045e02c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b87f7c8626eb4ec2838eb721ecfc6289'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:44.552367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-16a4256f-a812-4961-8aff-ca753045e02c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b87f7c8626eb4ec2838eb721ecfc6289'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:44.555784 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:44.574138 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 952/1924] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:44 2022] DELETE /identity/v3/users/b87f7c8626eb4ec2838eb721ecfc6289 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:44.858967 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:44.860626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:44.875408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd6e030-266c-4553-94bd-63796387a863 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:44.936864 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecca075c-8aa7-42de-91ef-b448ddeb389b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:44.937503 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecca075c-8aa7-42de-91ef-b448ddeb389b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:44.937780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecca075c-8aa7-42de-91ef-b448ddeb389b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:44.938064 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecca075c-8aa7-42de-91ef-b448ddeb389b admin admin] PATH_INFO: `/v3/projects/15694025b75d48fea0b2ab98a85cfb35` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:44.939179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecca075c-8aa7-42de-91ef-b448ddeb389b admin admin] RBAC: Authorizing `identity:delete_project(project_id=15694025b75d48fea0b2ab98a85cfb35)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:44.941111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecca075c-8aa7-42de-91ef-b448ddeb389b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:44.987780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ecca075c-8aa7-42de-91ef-b448ddeb389b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15694025b75d48fea0b2ab98a85cfb35', 'request_id': 'req-ecca075c-8aa7-42de-91ef-b448ddeb389b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:44.993193 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ecca075c-8aa7-42de-91ef-b448ddeb389b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15694025b75d48fea0b2ab98a85cfb35', 'request_id': 'req-ecca075c-8aa7-42de-91ef-b448ddeb389b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:45.000955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ecca075c-8aa7-42de-91ef-b448ddeb389b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15694025b75d48fea0b2ab98a85cfb35', 'request_id': 'req-ecca075c-8aa7-42de-91ef-b448ddeb389b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:45.011681 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ecca075c-8aa7-42de-91ef-b448ddeb389b admin admin] The token cache is being invalidate because project 15694025b75d48fea0b2ab98a85cfb35 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:45.011967 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ecca075c-8aa7-42de-91ef-b448ddeb389b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:45.012632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ecca075c-8aa7-42de-91ef-b448ddeb389b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:45.014660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 973/1925] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:44 2022] DELETE /identity/v3/projects/15694025b75d48fea0b2ab98a85cfb35 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:45.021723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6cf618-b820-4681-bcc5-a7d58f8ed261 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:45.022203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6cf618-b820-4681-bcc5-a7d58f8ed261 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:45.022293 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6cf618-b820-4681-bcc5-a7d58f8ed261 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:45.041852 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-eb6cf618-b820-4681-bcc5-a7d58f8ed261 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:45.043047 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-eb6cf618-b820-4681-bcc5-a7d58f8ed261 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:45.117546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 953/1926] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:45.119205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20113c42-4f28-4125-afb6-8d24332fde65 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:45.119496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20113c42-4f28-4125-afb6-8d24332fde65 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:45.119798 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20113c42-4f28-4125-afb6-8d24332fde65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:45.143351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-20113c42-4f28-4125-afb6-8d24332fde65 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:45.144782 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-20113c42-4f28-4125-afb6-8d24332fde65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:45.153351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 974/1927] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:45.158067 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6cf618-b820-4681-bcc5-a7d58f8ed261 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:45.170583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6cf618-b820-4681-bcc5-a7d58f8ed261 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:45.175892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caf2c6c-6e22-4881-94a1-003fcd51c9c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:45.176571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caf2c6c-6e22-4881-94a1-003fcd51c9c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:45.176878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caf2c6c-6e22-4881-94a1-003fcd51c9c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:45.177202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caf2c6c-6e22-4881-94a1-003fcd51c9c3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:45.177920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caf2c6c-6e22-4881-94a1-003fcd51c9c3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:15:45.178232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caf2c6c-6e22-4881-94a1-003fcd51c9c3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:45.180409 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caf2c6c-6e22-4881-94a1-003fcd51c9c3 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:45.198280 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 954/1928] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:15:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:15:45.204423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20113c42-4f28-4125-afb6-8d24332fde65 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:45.215973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20113c42-4f28-4125-afb6-8d24332fde65 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:45.221267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d85b05d-3958-4a4a-930f-2635bc930ec1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:45.221949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85b05d-3958-4a4a-930f-2635bc930ec1 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:45.222265 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85b05d-3958-4a4a-930f-2635bc930ec1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:45.222546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85b05d-3958-4a4a-930f-2635bc930ec1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:45.223216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d85b05d-3958-4a4a-930f-2635bc930ec1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:45.225211 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d85b05d-3958-4a4a-930f-2635bc930ec1 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:45.261188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:45.274755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 975/1929] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:45 2022] POST /identity/v3/projects => generated 373 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:45.277750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caf2c6c-6e22-4881-94a1-003fcd51c9c3 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:45.289204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caf2c6c-6e22-4881-94a1-003fcd51c9c3 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:45.371624 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1927a6c-ff05-42de-bff3-a21730746370 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:45.372953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1927a6c-ff05-42de-bff3-a21730746370 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:45.373263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1927a6c-ff05-42de-bff3-a21730746370 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:45.373538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1927a6c-ff05-42de-bff3-a21730746370 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:45.374175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1927a6c-ff05-42de-bff3-a21730746370 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:45.376925 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1927a6c-ff05-42de-bff3-a21730746370 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:45.426407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 955/1930] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:45 2022] POST /identity/v3/users => generated 287 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:45.427746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d85b05d-3958-4a4a-930f-2635bc930ec1 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:45.444190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d85b05d-3958-4a4a-930f-2635bc930ec1 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:45.478013 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a007d1ad-a287-46e2-bcfd-b104a84a6474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:45.478684 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a007d1ad-a287-46e2-bcfd-b104a84a6474 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:45.478901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a007d1ad-a287-46e2-bcfd-b104a84a6474 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:45.479170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a007d1ad-a287-46e2-bcfd-b104a84a6474 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:45.479889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a007d1ad-a287-46e2-bcfd-b104a84a6474 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:45.482255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a007d1ad-a287-46e2-bcfd-b104a84a6474 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:45.523193 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 976/1931] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:45 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:45.527228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1927a6c-ff05-42de-bff3-a21730746370 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:45.543125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1927a6c-ff05-42de-bff3-a21730746370 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:45.570691 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc77f586-b6da-4563-88e5-bdd045ebbc7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:45.571643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc77f586-b6da-4563-88e5-bdd045ebbc7b admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:45.572195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc77f586-b6da-4563-88e5-bdd045ebbc7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:45.572726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc77f586-b6da-4563-88e5-bdd045ebbc7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:45.573453 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc77f586-b6da-4563-88e5-bdd045ebbc7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:45.575320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc77f586-b6da-4563-88e5-bdd045ebbc7b admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:45.596348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 956/1932] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:45 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:45.598651 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a007d1ad-a287-46e2-bcfd-b104a84a6474 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:45.609230 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a007d1ad-a287-46e2-bcfd-b104a84a6474 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:45.614580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7e1121-aa59-4061-aa18-167e1ed206b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:45.615620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7e1121-aa59-4061-aa18-167e1ed206b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:45.615827 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7e1121-aa59-4061-aa18-167e1ed206b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:45.616204 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7e1121-aa59-4061-aa18-167e1ed206b5 admin admin] PATH_INFO: `/v3/projects/4eec2247c00d4f5997b19aea1ff8a57d/users/d825779319cb467686199635c1c04afd/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:45.640126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7e1121-aa59-4061-aa18-167e1ed206b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4eec2247c00d4f5997b19aea1ff8a57d, user_id=d825779319cb467686199635c1c04afd, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:45.643203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7e1121-aa59-4061-aa18-167e1ed206b5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:45.672855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 977/1933] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:45 2022] PUT /identity/v3/projects/4eec2247c00d4f5997b19aea1ff8a57d/users/d825779319cb467686199635c1c04afd/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:15:45.684451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba45cda-fff4-44a4-81b9-d78f439545bb None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:45.685253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba45cda-fff4-44a4-81b9-d78f439545bb None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:45.685542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba45cda-fff4-44a4-81b9-d78f439545bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:45.744765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-0ba45cda-fff4-44a4-81b9-d78f439545bb None None] MFA Rules not processed for user `d825779319cb467686199635c1c04afd`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:45.746078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0ba45cda-fff4-44a4-81b9-d78f439545bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:45.823315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:DELETE=1,keystone:SELECT=96,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:45.850420 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 957/1934] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:45 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:45.878400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7e1121-aa59-4061-aa18-167e1ed206b5 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:45.881266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ca7e1121-aa59-4061-aa18-167e1ed206b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:45.947294 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7e1121-aa59-4061-aa18-167e1ed206b5 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:45.951617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df1dadc-648e-42fe-958f-f27c8ee0ba2a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:45.952258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df1dadc-648e-42fe-958f-f27c8ee0ba2a service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:45.952602 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df1dadc-648e-42fe-958f-f27c8ee0ba2a service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:45.952920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df1dadc-648e-42fe-958f-f27c8ee0ba2a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:45.967118 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df1dadc-648e-42fe-958f-f27c8ee0ba2a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:45.968739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df1dadc-648e-42fe-958f-f27c8ee0ba2a service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:45.978627 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 978/1935] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:45 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:46.005520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba45cda-fff4-44a4-81b9-d78f439545bb None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:46.092524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba45cda-fff4-44a4-81b9-d78f439545bb None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:46.097422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8a294c-2b48-4826-bf92-1120f0c55271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:46.098029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8a294c-2b48-4826-bf92-1120f0c55271 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:46.098452 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8a294c-2b48-4826-bf92-1120f0c55271 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:46.098996 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8a294c-2b48-4826-bf92-1120f0c55271 admin admin] PATH_INFO: `/v3/users/d825779319cb467686199635c1c04afd` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:46.100134 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8a294c-2b48-4826-bf92-1120f0c55271 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d825779319cb467686199635c1c04afd)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:46.102085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8a294c-2b48-4826-bf92-1120f0c55271 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:46.149759 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fb8a294c-2b48-4826-bf92-1120f0c55271 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd825779319cb467686199635c1c04afd'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:46.150504 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fb8a294c-2b48-4826-bf92-1120f0c55271 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd825779319cb467686199635c1c04afd'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:46.151145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fb8a294c-2b48-4826-bf92-1120f0c55271 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd825779319cb467686199635c1c04afd'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:46.159035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fb8a294c-2b48-4826-bf92-1120f0c55271 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd825779319cb467686199635c1c04afd'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:46.172008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fb8a294c-2b48-4826-bf92-1120f0c55271 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd825779319cb467686199635c1c04afd'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:46.185818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 958/1936] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:46 2022] DELETE /identity/v3/users/d825779319cb467686199635c1c04afd => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:46.190401 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf66eb9d-5c8f-4ade-b3c2-76a7e73b95a0 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:46.190691 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf66eb9d-5c8f-4ade-b3c2-76a7e73b95a0 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:46.190990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf66eb9d-5c8f-4ade-b3c2-76a7e73b95a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:46.210919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-bf66eb9d-5c8f-4ade-b3c2-76a7e73b95a0 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:46.212121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-bf66eb9d-5c8f-4ade-b3c2-76a7e73b95a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:46.289199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 979/1937] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:46.298840 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8a294c-2b48-4826-bf92-1120f0c55271 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:46.300288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-fb8a294c-2b48-4826-bf92-1120f0c55271 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:46.386315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8a294c-2b48-4826-bf92-1120f0c55271 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:46.396297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f45578-d577-4dbb-93f7-867c72468d13 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:46.396297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f45578-d577-4dbb-93f7-867c72468d13 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:46.396297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f45578-d577-4dbb-93f7-867c72468d13 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:46.396297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f45578-d577-4dbb-93f7-867c72468d13 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:46.423271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f45578-d577-4dbb-93f7-867c72468d13 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:46.424770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f45578-d577-4dbb-93f7-867c72468d13 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:46.441057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 959/1938] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:46.912438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf66eb9d-5c8f-4ade-b3c2-76a7e73b95a0 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:46.914028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-bf66eb9d-5c8f-4ade-b3c2-76a7e73b95a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:46.934617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf66eb9d-5c8f-4ade-b3c2-76a7e73b95a0 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:46.949682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e0f5bc-30ba-4000-8ef9-491a5259500e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:46.950874 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e0f5bc-30ba-4000-8ef9-491a5259500e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:46.951153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e0f5bc-30ba-4000-8ef9-491a5259500e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:46.951422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e0f5bc-30ba-4000-8ef9-491a5259500e admin admin] PATH_INFO: `/v3/projects/4eec2247c00d4f5997b19aea1ff8a57d` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:46.955719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e0f5bc-30ba-4000-8ef9-491a5259500e admin admin] RBAC: Authorizing `identity:delete_project(project_id=4eec2247c00d4f5997b19aea1ff8a57d)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:46.957870 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e0f5bc-30ba-4000-8ef9-491a5259500e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:47.067336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-83e0f5bc-30ba-4000-8ef9-491a5259500e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4eec2247c00d4f5997b19aea1ff8a57d', 'request_id': 'req-83e0f5bc-30ba-4000-8ef9-491a5259500e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:47.073369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-83e0f5bc-30ba-4000-8ef9-491a5259500e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4eec2247c00d4f5997b19aea1ff8a57d', 'request_id': 'req-83e0f5bc-30ba-4000-8ef9-491a5259500e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:47.076055 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:47.082683 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-83e0f5bc-30ba-4000-8ef9-491a5259500e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4eec2247c00d4f5997b19aea1ff8a57d', 'request_id': 'req-83e0f5bc-30ba-4000-8ef9-491a5259500e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:47.097612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-83e0f5bc-30ba-4000-8ef9-491a5259500e admin admin] The token cache is being invalidate because project 4eec2247c00d4f5997b19aea1ff8a57d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:47.097961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-83e0f5bc-30ba-4000-8ef9-491a5259500e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:47.098629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-83e0f5bc-30ba-4000-8ef9-491a5259500e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:47.103467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 980/1939] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:46 2022] DELETE /identity/v3/projects/4eec2247c00d4f5997b19aea1ff8a57d => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:47.105091 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3243c3-a7df-4669-ab44-693ecc0983cb None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:47.105559 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3243c3-a7df-4669-ab44-693ecc0983cb None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:47.105656 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3243c3-a7df-4669-ab44-693ecc0983cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:47.131982 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-9f3243c3-a7df-4669-ab44-693ecc0983cb None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:47.133521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9f3243c3-a7df-4669-ab44-693ecc0983cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:47.142184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:47.223973 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 960/1940] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:47.232530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d2d880-204f-4896-a776-4e21689166e5 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:47.232873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d2d880-204f-4896-a776-4e21689166e5 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:47.233202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d2d880-204f-4896-a776-4e21689166e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:47.253130 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-73d2d880-204f-4896-a776-4e21689166e5 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:47.254405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-73d2d880-204f-4896-a776-4e21689166e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:47.265094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 981/1941] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:47.269207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3243c3-a7df-4669-ab44-693ecc0983cb None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:47.279432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3243c3-a7df-4669-ab44-693ecc0983cb None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:47.283993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f767f0a5-fa13-479e-8690-84eb18e8b009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:47.284676 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f767f0a5-fa13-479e-8690-84eb18e8b009 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:47.284930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f767f0a5-fa13-479e-8690-84eb18e8b009 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:47.285197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f767f0a5-fa13-479e-8690-84eb18e8b009 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:47.285859 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f767f0a5-fa13-479e-8690-84eb18e8b009 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:15:47.286157 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f767f0a5-fa13-479e-8690-84eb18e8b009 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:47.287815 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f767f0a5-fa13-479e-8690-84eb18e8b009 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:47.300100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 961/1942] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:15:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:15:47.304362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d2d880-204f-4896-a776-4e21689166e5 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:47.313464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d2d880-204f-4896-a776-4e21689166e5 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:47.318087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f47d24-ec2e-4305-9552-bb95db2377b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:47.318680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f47d24-ec2e-4305-9552-bb95db2377b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:47.318958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f47d24-ec2e-4305-9552-bb95db2377b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:47.319305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f47d24-ec2e-4305-9552-bb95db2377b4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:47.319857 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f47d24-ec2e-4305-9552-bb95db2377b4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:47.321942 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f47d24-ec2e-4305-9552-bb95db2377b4 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:47.355887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 982/1943] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:47 2022] POST /identity/v3/projects => generated 375 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:47.357662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f767f0a5-fa13-479e-8690-84eb18e8b009 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:47.369937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f767f0a5-fa13-479e-8690-84eb18e8b009 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:47.453336 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f220f4a5-bb8d-4554-9b5d-ffd6d39f7299 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:47.454113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f220f4a5-bb8d-4554-9b5d-ffd6d39f7299 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:47.454523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f220f4a5-bb8d-4554-9b5d-ffd6d39f7299 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:47.454890 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f220f4a5-bb8d-4554-9b5d-ffd6d39f7299 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:47.455638 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f220f4a5-bb8d-4554-9b5d-ffd6d39f7299 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:47.457741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f220f4a5-bb8d-4554-9b5d-ffd6d39f7299 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:47.485341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 962/1944] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:47 2022] POST /identity/v3/users => generated 288 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:47.489802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f47d24-ec2e-4305-9552-bb95db2377b4 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:47.503359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f47d24-ec2e-4305-9552-bb95db2377b4 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:47.520438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39252537-96b5-4d45-9041-1b72094ed88b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:47.520438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39252537-96b5-4d45-9041-1b72094ed88b admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:47.520438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39252537-96b5-4d45-9041-1b72094ed88b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:47.520438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39252537-96b5-4d45-9041-1b72094ed88b admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:47.520809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39252537-96b5-4d45-9041-1b72094ed88b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:47.520809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39252537-96b5-4d45-9041-1b72094ed88b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:47.526564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 983/1945] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:47 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:47.531317 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f220f4a5-bb8d-4554-9b5d-ffd6d39f7299 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:47.543674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f220f4a5-bb8d-4554-9b5d-ffd6d39f7299 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:47.549781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4753e763-31b6-4552-9bf5-1e3af820a887 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:47.550500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4753e763-31b6-4552-9bf5-1e3af820a887 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:47.550887 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4753e763-31b6-4552-9bf5-1e3af820a887 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:47.551836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4753e763-31b6-4552-9bf5-1e3af820a887 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:47.552795 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4753e763-31b6-4552-9bf5-1e3af820a887 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:47.554692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4753e763-31b6-4552-9bf5-1e3af820a887 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:47.574625 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39252537-96b5-4d45-9041-1b72094ed88b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:47.585402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 963/1946] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:47 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:47.587837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39252537-96b5-4d45-9041-1b72094ed88b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:47.595019 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e115dbc4-d89c-43de-98f2-60d6258e777c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:47.595806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e115dbc4-d89c-43de-98f2-60d6258e777c admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:47.596188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e115dbc4-d89c-43de-98f2-60d6258e777c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:47.596683 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e115dbc4-d89c-43de-98f2-60d6258e777c admin admin] PATH_INFO: `/v3/projects/0b10e28fddf44dcc9f11cdd65b178bdc/users/3599e0341d2848d2b9524ae28fd789b7/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:47.620554 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e115dbc4-d89c-43de-98f2-60d6258e777c admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b10e28fddf44dcc9f11cdd65b178bdc, user_id=3599e0341d2848d2b9524ae28fd789b7, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:47.623378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e115dbc4-d89c-43de-98f2-60d6258e777c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:47.648207 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 984/1947] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:47 2022] PUT /identity/v3/projects/0b10e28fddf44dcc9f11cdd65b178bdc/users/3599e0341d2848d2b9524ae28fd789b7/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:47.660916 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3526d570-993e-4a89-a9d8-1a813c69979d None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:47.661273 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3526d570-993e-4a89-a9d8-1a813c69979d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:47.661626 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3526d570-993e-4a89-a9d8-1a813c69979d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:47.720878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3526d570-993e-4a89-a9d8-1a813c69979d None None] MFA Rules not processed for user `3599e0341d2848d2b9524ae28fd789b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:47.722488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3526d570-993e-4a89-a9d8-1a813c69979d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:47.836536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 964/1948] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:47 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:47.842881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e115dbc4-d89c-43de-98f2-60d6258e777c admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:47.891273 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3526d570-993e-4a89-a9d8-1a813c69979d None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:47.893354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3526d570-993e-4a89-a9d8-1a813c69979d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:47.904585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:47.909478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e115dbc4-d89c-43de-98f2-60d6258e777c admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:47.915256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e6624f-bce8-4970-b536-80c59a8ad5f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:47.916053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e6624f-bce8-4970-b536-80c59a8ad5f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:47.916440 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e6624f-bce8-4970-b536-80c59a8ad5f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:47.916776 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e6624f-bce8-4970-b536-80c59a8ad5f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:47.917479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e6624f-bce8-4970-b536-80c59a8ad5f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:47.919493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e6624f-bce8-4970-b536-80c59a8ad5f9 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:47.952355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 985/1949] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:47 2022] POST /identity/v3/projects => generated 373 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:47.956855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e6624f-bce8-4970-b536-80c59a8ad5f9 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:47.976529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3526d570-993e-4a89-a9d8-1a813c69979d None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:48.050365 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c493ed73-37b7-4492-a79f-bd7fc9cdef55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:48.052117 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c493ed73-37b7-4492-a79f-bd7fc9cdef55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:48.052554 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c493ed73-37b7-4492-a79f-bd7fc9cdef55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:48.053166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c493ed73-37b7-4492-a79f-bd7fc9cdef55 admin admin] PATH_INFO: `/v3/users/cdadc0b0cde44cfabe4cba6103a59842` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:48.054856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c493ed73-37b7-4492-a79f-bd7fc9cdef55 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cdadc0b0cde44cfabe4cba6103a59842)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:48.056429 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e6624f-bce8-4970-b536-80c59a8ad5f9 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:48.057247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c493ed73-37b7-4492-a79f-bd7fc9cdef55 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:48.063284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf18c47e-a4e1-45b8-9eb9-1102a9bb5edf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:48.064605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf18c47e-a4e1-45b8-9eb9-1102a9bb5edf admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:48.065057 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf18c47e-a4e1-45b8-9eb9-1102a9bb5edf admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:48.065492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf18c47e-a4e1-45b8-9eb9-1102a9bb5edf admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:48.066773 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf18c47e-a4e1-45b8-9eb9-1102a9bb5edf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:48.069380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf18c47e-a4e1-45b8-9eb9-1102a9bb5edf admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:48.093925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:DELETE=1,keystone:SELECT=93,keystone:INSERT=6 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:48.110166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 986/1950] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:47 2022] POST /identity/v3/users => generated 293 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:48.111703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf18c47e-a4e1-45b8-9eb9-1102a9bb5edf admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:48.132563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf18c47e-a4e1-45b8-9eb9-1102a9bb5edf admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:48.146409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba56254f-e5e9-4c4e-87e6-2f2729d32aad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:48.148436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba56254f-e5e9-4c4e-87e6-2f2729d32aad admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:48.148914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba56254f-e5e9-4c4e-87e6-2f2729d32aad admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:48.149297 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba56254f-e5e9-4c4e-87e6-2f2729d32aad admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:48.150206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba56254f-e5e9-4c4e-87e6-2f2729d32aad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:48.152288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba56254f-e5e9-4c4e-87e6-2f2729d32aad admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:48.166479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 987/1951] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:48 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:48.173297 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba56254f-e5e9-4c4e-87e6-2f2729d32aad admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:48.177132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c493ed73-37b7-4492-a79f-bd7fc9cdef55 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cdadc0b0cde44cfabe4cba6103a59842'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:48.178264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c493ed73-37b7-4492-a79f-bd7fc9cdef55 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cdadc0b0cde44cfabe4cba6103a59842'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:48.179665 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c493ed73-37b7-4492-a79f-bd7fc9cdef55 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cdadc0b0cde44cfabe4cba6103a59842'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:48.185589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba56254f-e5e9-4c4e-87e6-2f2729d32aad admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:48.191818 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be738d1-3c38-495d-b45b-d8a35520333b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:48.193219 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be738d1-3c38-495d-b45b-d8a35520333b admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:48.193825 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be738d1-3c38-495d-b45b-d8a35520333b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:48.194446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be738d1-3c38-495d-b45b-d8a35520333b admin admin] PATH_INFO: `/v3/projects/ec2219d8b0b34381a22658dfda92b8e1/users/e439ef3b1e7d47cca73cdfe67e8bb253/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:48.196599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c493ed73-37b7-4492-a79f-bd7fc9cdef55 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cdadc0b0cde44cfabe4cba6103a59842'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:48.220404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be738d1-3c38-495d-b45b-d8a35520333b admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec2219d8b0b34381a22658dfda92b8e1, user_id=e439ef3b1e7d47cca73cdfe67e8bb253, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:48.223564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be738d1-3c38-495d-b45b-d8a35520333b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:48.227192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c493ed73-37b7-4492-a79f-bd7fc9cdef55 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cdadc0b0cde44cfabe4cba6103a59842'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:48.245211 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 988/1952] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:48 2022] PUT /identity/v3/projects/ec2219d8b0b34381a22658dfda92b8e1/users/e439ef3b1e7d47cca73cdfe67e8bb253/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:48.251113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 965/1953] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:47 2022] DELETE /identity/v3/users/cdadc0b0cde44cfabe4cba6103a59842 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:48.252389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be738d1-3c38-495d-b45b-d8a35520333b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:48.255730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cba91d-aff6-4154-a52c-632a45d7162c None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:48.256149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cba91d-aff6-4154-a52c-632a45d7162c None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:48.256725 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cba91d-aff6-4154-a52c-632a45d7162c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:48.260977 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2be738d1-3c38-495d-b45b-d8a35520333b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:48.289426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-19cba91d-aff6-4154-a52c-632a45d7162c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:48.290607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-19cba91d-aff6-4154-a52c-632a45d7162c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:48.333341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:48.384509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 966/1954] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:48.388482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be738d1-3c38-495d-b45b-d8a35520333b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:48.391591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050eecf5-7205-49a9-be93-56acbcabfade admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:48.392344 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050eecf5-7205-49a9-be93-56acbcabfade admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:48.392693 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050eecf5-7205-49a9-be93-56acbcabfade admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:48.393032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050eecf5-7205-49a9-be93-56acbcabfade admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:48.393740 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050eecf5-7205-49a9-be93-56acbcabfade admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:48.395090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cba91d-aff6-4154-a52c-632a45d7162c None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:48.395388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050eecf5-7205-49a9-be93-56acbcabfade admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:48.396960 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-19cba91d-aff6-4154-a52c-632a45d7162c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:48.408056 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 989/1955] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:48 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:48.411939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050eecf5-7205-49a9-be93-56acbcabfade admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:48.422734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050eecf5-7205-49a9-be93-56acbcabfade admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:48.427725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56983966-eda6-4c5a-b3ca-3bfb28316971 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:48.428611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56983966-eda6-4c5a-b3ca-3bfb28316971 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:48.429280 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56983966-eda6-4c5a-b3ca-3bfb28316971 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:48.429560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56983966-eda6-4c5a-b3ca-3bfb28316971 admin admin] PATH_INFO: `/v3/projects/ec2219d8b0b34381a22658dfda92b8e1/users/e439ef3b1e7d47cca73cdfe67e8bb253/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:48.431804 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56983966-eda6-4c5a-b3ca-3bfb28316971 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec2219d8b0b34381a22658dfda92b8e1, user_id=e439ef3b1e7d47cca73cdfe67e8bb253, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:48.434687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56983966-eda6-4c5a-b3ca-3bfb28316971 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:48.446251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 990/1956] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:48 2022] PUT /identity/v3/projects/ec2219d8b0b34381a22658dfda92b8e1/users/e439ef3b1e7d47cca73cdfe67e8bb253/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:15:48.462094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efc04aa-9150-439b-8d01-a70ae4cd9a05 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:48.462680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efc04aa-9150-439b-8d01-a70ae4cd9a05 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:48.462956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efc04aa-9150-439b-8d01-a70ae4cd9a05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:48.499072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cba91d-aff6-4154-a52c-632a45d7162c None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:48.505023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-5efc04aa-9150-439b-8d01-a70ae4cd9a05 None None] MFA Rules not processed for user `e439ef3b1e7d47cca73cdfe67e8bb253`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:48.506486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5efc04aa-9150-439b-8d01-a70ae4cd9a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:48.572125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a723720b-92fb-49dc-b709-43d8defb6b8f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:48.572891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a723720b-92fb-49dc-b709-43d8defb6b8f service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:48.573222 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a723720b-92fb-49dc-b709-43d8defb6b8f service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:48.573585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a723720b-92fb-49dc-b709-43d8defb6b8f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:48.638833 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 991/1957] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:48 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:48.650631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a723720b-92fb-49dc-b709-43d8defb6b8f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:48.652168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a723720b-92fb-49dc-b709-43d8defb6b8f service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:48.655043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efc04aa-9150-439b-8d01-a70ae4cd9a05 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:48.656659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5efc04aa-9150-439b-8d01-a70ae4cd9a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:48.676616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 967/1958] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:15:48.703666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:48.734436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efc04aa-9150-439b-8d01-a70ae4cd9a05 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:48.739313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09aaebb2-ccc8-48d0-aca8-301647718c38 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:48.740040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aaebb2-ccc8-48d0-aca8-301647718c38 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:48.740406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aaebb2-ccc8-48d0-aca8-301647718c38 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:48.741016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aaebb2-ccc8-48d0-aca8-301647718c38 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:48.744712 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-09aaebb2-ccc8-48d0-aca8-301647718c38 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:48.874676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09aaebb2-ccc8-48d0-aca8-301647718c38 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:48.879947 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09aaebb2-ccc8-48d0-aca8-301647718c38 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:48.920648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 992/1959] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:48 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:49.295964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a723720b-92fb-49dc-b709-43d8defb6b8f service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:49.297447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a723720b-92fb-49dc-b709-43d8defb6b8f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:49.318166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a723720b-92fb-49dc-b709-43d8defb6b8f service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:49.328898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:49.329550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:49.329828 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:49.330098 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] PATH_INFO: `/v3/projects/7e9f82c2dd784c2a837ec44457201aad` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:49.331239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e9f82c2dd784c2a837ec44457201aad)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:49.338254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:49.435391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:49.452704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e9f82c2dd784c2a837ec44457201aad', 'request_id': 'req-462e3c95-7de4-4509-9359-d69857d4c064'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:49.464728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e9f82c2dd784c2a837ec44457201aad', 'request_id': 'req-462e3c95-7de4-4509-9359-d69857d4c064'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:49.476377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e9f82c2dd784c2a837ec44457201aad', 'request_id': 'req-462e3c95-7de4-4509-9359-d69857d4c064'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:49.501680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] The token cache is being invalidate because project 7e9f82c2dd784c2a837ec44457201aad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:49.501977 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:49.502712 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:49.504501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 968/1960] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:49 2022] DELETE /identity/v3/projects/7e9f82c2dd784c2a837ec44457201aad => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:49.508480 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09aaebb2-ccc8-48d0-aca8-301647718c38 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:49.522093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09aaebb2-ccc8-48d0-aca8-301647718c38 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:49.631740 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47deb654-d729-4ce3-be3d-298de18e24d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:49.632217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47deb654-d729-4ce3-be3d-298de18e24d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:49.632536 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47deb654-d729-4ce3-be3d-298de18e24d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:49.632835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47deb654-d729-4ce3-be3d-298de18e24d5 admin admin] PATH_INFO: `/v3/users/d86be4405e5842558bef13938bff29c8` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:49.633959 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47deb654-d729-4ce3-be3d-298de18e24d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d86be4405e5842558bef13938bff29c8)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:49.635897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47deb654-d729-4ce3-be3d-298de18e24d5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:49.692992 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-47deb654-d729-4ce3-be3d-298de18e24d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd86be4405e5842558bef13938bff29c8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:49.693735 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-47deb654-d729-4ce3-be3d-298de18e24d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd86be4405e5842558bef13938bff29c8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:49.703595 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-47deb654-d729-4ce3-be3d-298de18e24d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd86be4405e5842558bef13938bff29c8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:49.704265 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-47deb654-d729-4ce3-be3d-298de18e24d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd86be4405e5842558bef13938bff29c8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:49.723807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-47deb654-d729-4ce3-be3d-298de18e24d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd86be4405e5842558bef13938bff29c8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:49.741154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 993/1961] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:49 2022] DELETE /identity/v3/users/d86be4405e5842558bef13938bff29c8 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:15:50.162927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:50.164844 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:50.181704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462e3c95-7de4-4509-9359-d69857d4c064 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:50.271907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f063270-c96f-42b5-a542-defc70b9faf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:50.272585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f063270-c96f-42b5-a542-defc70b9faf7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:50.272920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f063270-c96f-42b5-a542-defc70b9faf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:50.273203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f063270-c96f-42b5-a542-defc70b9faf7 admin admin] PATH_INFO: `/v3/projects/8c78391efde441cd81b1fd99a56e45c9` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:50.274436 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f063270-c96f-42b5-a542-defc70b9faf7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8c78391efde441cd81b1fd99a56e45c9)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:50.276601 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f063270-c96f-42b5-a542-defc70b9faf7 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:50.349999 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6f063270-c96f-42b5-a542-defc70b9faf7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8c78391efde441cd81b1fd99a56e45c9', 'request_id': 'req-6f063270-c96f-42b5-a542-defc70b9faf7'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:50.364180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6f063270-c96f-42b5-a542-defc70b9faf7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8c78391efde441cd81b1fd99a56e45c9', 'request_id': 'req-6f063270-c96f-42b5-a542-defc70b9faf7'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:50.377611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6f063270-c96f-42b5-a542-defc70b9faf7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8c78391efde441cd81b1fd99a56e45c9', 'request_id': 'req-6f063270-c96f-42b5-a542-defc70b9faf7'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:50.421221 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6f063270-c96f-42b5-a542-defc70b9faf7 admin admin] The token cache is being invalidate because project 8c78391efde441cd81b1fd99a56e45c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:50.421864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6f063270-c96f-42b5-a542-defc70b9faf7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:50.423785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6f063270-c96f-42b5-a542-defc70b9faf7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:50.426680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 969/1962] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:50 2022] DELETE /identity/v3/projects/8c78391efde441cd81b1fd99a56e45c9 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:50.432826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3ca461-4917-4a69-a2bc-522f02d89932 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:50.433325 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3ca461-4917-4a69-a2bc-522f02d89932 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:50.433612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3ca461-4917-4a69-a2bc-522f02d89932 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:50.492630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:50.510595 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-ff3ca461-4917-4a69-a2bc-522f02d89932 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:50.511821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ff3ca461-4917-4a69-a2bc-522f02d89932 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:50.576250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 994/1963] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:50.584785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504dfa6d-e302-4acb-b0dc-4b205028a009 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:50.585280 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504dfa6d-e302-4acb-b0dc-4b205028a009 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:50.585661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504dfa6d-e302-4acb-b0dc-4b205028a009 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:50.607856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-504dfa6d-e302-4acb-b0dc-4b205028a009 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:50.609576 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-504dfa6d-e302-4acb-b0dc-4b205028a009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:50.631630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3ca461-4917-4a69-a2bc-522f02d89932 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:50.649781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 970/1964] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:50.659116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3ca461-4917-4a69-a2bc-522f02d89932 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:50.678179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919fbfdf-dae4-494d-bdee-1ee3620e2df5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:50.681650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919fbfdf-dae4-494d-bdee-1ee3620e2df5 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:50.682203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919fbfdf-dae4-494d-bdee-1ee3620e2df5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:50.682680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919fbfdf-dae4-494d-bdee-1ee3620e2df5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:50.683632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919fbfdf-dae4-494d-bdee-1ee3620e2df5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:15:50.685883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919fbfdf-dae4-494d-bdee-1ee3620e2df5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:50.692554 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919fbfdf-dae4-494d-bdee-1ee3620e2df5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:50.717333 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 995/1965] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:15:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:15:50.722567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504dfa6d-e302-4acb-b0dc-4b205028a009 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:50.751755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504dfa6d-e302-4acb-b0dc-4b205028a009 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:50.773458 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41fb7ab-61fb-4f7a-aec9-2b151194db67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:50.774271 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41fb7ab-61fb-4f7a-aec9-2b151194db67 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:50.774666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41fb7ab-61fb-4f7a-aec9-2b151194db67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:50.775071 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41fb7ab-61fb-4f7a-aec9-2b151194db67 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:50.775974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41fb7ab-61fb-4f7a-aec9-2b151194db67 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:50.779140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41fb7ab-61fb-4f7a-aec9-2b151194db67 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:50.826128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 971/1966] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:50 2022] POST /identity/v3/projects => generated 383 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:50.830881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919fbfdf-dae4-494d-bdee-1ee3620e2df5 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:50.862112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919fbfdf-dae4-494d-bdee-1ee3620e2df5 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:50.937009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692a7389-1533-4d6f-b849-12466fc255d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:50.937634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692a7389-1533-4d6f-b849-12466fc255d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:50.937907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692a7389-1533-4d6f-b849-12466fc255d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:50.938177 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692a7389-1533-4d6f-b849-12466fc255d0 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:50.938810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692a7389-1533-4d6f-b849-12466fc255d0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:50.940948 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692a7389-1533-4d6f-b849-12466fc255d0 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:50.963437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 996/1967] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:50 2022] POST /identity/v3/users => generated 292 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:50.967890 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41fb7ab-61fb-4f7a-aec9-2b151194db67 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:50.979329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41fb7ab-61fb-4f7a-aec9-2b151194db67 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:50.984128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b53b0a0-a7ca-4997-b45d-ed53862a23f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:50.985412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b53b0a0-a7ca-4997-b45d-ed53862a23f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:50.985699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b53b0a0-a7ca-4997-b45d-ed53862a23f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:50.985984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b53b0a0-a7ca-4997-b45d-ed53862a23f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:50.987176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b53b0a0-a7ca-4997-b45d-ed53862a23f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:50.988820 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b53b0a0-a7ca-4997-b45d-ed53862a23f5 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:50.999681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 972/1968] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:50 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:51.003739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692a7389-1533-4d6f-b849-12466fc255d0 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:51.015116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692a7389-1533-4d6f-b849-12466fc255d0 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:51.019900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8406a6-15b5-49c2-8a65-ae8c288745b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:51.020676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8406a6-15b5-49c2-8a65-ae8c288745b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:51.021021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8406a6-15b5-49c2-8a65-ae8c288745b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:51.021803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8406a6-15b5-49c2-8a65-ae8c288745b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:51.021929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8406a6-15b5-49c2-8a65-ae8c288745b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:51.023549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8406a6-15b5-49c2-8a65-ae8c288745b5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:51.034975 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 997/1969] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:51.039001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b53b0a0-a7ca-4997-b45d-ed53862a23f5 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:51.049470 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b53b0a0-a7ca-4997-b45d-ed53862a23f5 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:51.055022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0451fa6-09ea-46bc-8eff-fa3cb60c496b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:51.055676 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0451fa6-09ea-46bc-8eff-fa3cb60c496b admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:51.055951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0451fa6-09ea-46bc-8eff-fa3cb60c496b admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:51.056221 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0451fa6-09ea-46bc-8eff-fa3cb60c496b admin admin] PATH_INFO: `/v3/projects/258d9e33e4d54f9894f1d54c53e3b4ba/users/9bb923e0d3fb49cfb58b8b4329fedae4/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:51.068367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:51.073234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0451fa6-09ea-46bc-8eff-fa3cb60c496b admin admin] RBAC: Authorizing `identity:create_grant(project_id=258d9e33e4d54f9894f1d54c53e3b4ba, user_id=9bb923e0d3fb49cfb58b8b4329fedae4, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:51.076716 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0451fa6-09ea-46bc-8eff-fa3cb60c496b admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:51.091168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 973/1970] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:51 2022] PUT /identity/v3/projects/258d9e33e4d54f9894f1d54c53e3b4ba/users/9bb923e0d3fb49cfb58b8b4329fedae4/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:51.102660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ab27b2-5f27-4cd0-9b36-74665a44ee4a None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:51.102949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ab27b2-5f27-4cd0-9b36-74665a44ee4a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:51.103227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ab27b2-5f27-4cd0-9b36-74665a44ee4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:51.133621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-d0ab27b2-5f27-4cd0-9b36-74665a44ee4a None None] MFA Rules not processed for user `9bb923e0d3fb49cfb58b8b4329fedae4`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:51.134836 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d0ab27b2-5f27-4cd0-9b36-74665a44ee4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:51.204682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 998/1971] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:51 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:51.217245 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0451fa6-09ea-46bc-8eff-fa3cb60c496b admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:51.218951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a0451fa6-09ea-46bc-8eff-fa3cb60c496b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:51.278950 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0451fa6-09ea-46bc-8eff-fa3cb60c496b admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:51.282726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65c0f36-106e-4e7c-bcdc-892ccc37650e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:51.283853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c0f36-106e-4e7c-bcdc-892ccc37650e service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:51.283853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c0f36-106e-4e7c-bcdc-892ccc37650e service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:51.283853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c0f36-106e-4e7c-bcdc-892ccc37650e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:51.292862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65c0f36-106e-4e7c-bcdc-892ccc37650e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:51.296408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65c0f36-106e-4e7c-bcdc-892ccc37650e service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:51.305069 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 974/1972] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:51 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:51.766417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ab27b2-5f27-4cd0-9b36-74665a44ee4a None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:51.767838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d0ab27b2-5f27-4cd0-9b36-74665a44ee4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:51.772887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:51.834033 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ab27b2-5f27-4cd0-9b36-74665a44ee4a None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:51.838492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dc396e-d30f-44b0-b7dc-45f39d54e514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:51.839108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dc396e-d30f-44b0-b7dc-45f39d54e514 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:51.839384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dc396e-d30f-44b0-b7dc-45f39d54e514 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:51.839655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dc396e-d30f-44b0-b7dc-45f39d54e514 admin admin] PATH_INFO: `/v3/users/3599e0341d2848d2b9524ae28fd789b7` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:51.840829 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47dc396e-d30f-44b0-b7dc-45f39d54e514 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3599e0341d2848d2b9524ae28fd789b7)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:51.842722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47dc396e-d30f-44b0-b7dc-45f39d54e514 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:51.892121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-47dc396e-d30f-44b0-b7dc-45f39d54e514 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3599e0341d2848d2b9524ae28fd789b7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:51.892843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-47dc396e-d30f-44b0-b7dc-45f39d54e514 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3599e0341d2848d2b9524ae28fd789b7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:51.900843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-47dc396e-d30f-44b0-b7dc-45f39d54e514 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3599e0341d2848d2b9524ae28fd789b7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:51.901765 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-47dc396e-d30f-44b0-b7dc-45f39d54e514 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3599e0341d2848d2b9524ae28fd789b7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:51.924791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-47dc396e-d30f-44b0-b7dc-45f39d54e514 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3599e0341d2848d2b9524ae28fd789b7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:51.948383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 999/1973] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:51 2022] DELETE /identity/v3/users/3599e0341d2848d2b9524ae28fd789b7 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:51.950072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb76ddf2-5529-4193-97e6-038ad7cb4b20 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:51.950670 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb76ddf2-5529-4193-97e6-038ad7cb4b20 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:51.950670 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb76ddf2-5529-4193-97e6-038ad7cb4b20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:51.976477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-bb76ddf2-5529-4193-97e6-038ad7cb4b20 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:51.978037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-bb76ddf2-5529-4193-97e6-038ad7cb4b20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:52.062537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 975/1974] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:52.075833 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dc396e-d30f-44b0-b7dc-45f39d54e514 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:52.077271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-47dc396e-d30f-44b0-b7dc-45f39d54e514 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:52.134429 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dc396e-d30f-44b0-b7dc-45f39d54e514 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:52.138493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d771250a-074a-4fcd-bd0a-70d89e4e8358 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:52.139087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d771250a-074a-4fcd-bd0a-70d89e4e8358 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:52.139367 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d771250a-074a-4fcd-bd0a-70d89e4e8358 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:52.139640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d771250a-074a-4fcd-bd0a-70d89e4e8358 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:52.148534 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d771250a-074a-4fcd-bd0a-70d89e4e8358 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:52.149963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d771250a-074a-4fcd-bd0a-70d89e4e8358 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:52.160227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1000/1975] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:52.438248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb76ddf2-5529-4193-97e6-038ad7cb4b20 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:52.439795 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-bb76ddf2-5529-4193-97e6-038ad7cb4b20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:52.455990 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb76ddf2-5529-4193-97e6-038ad7cb4b20 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:52.461107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:52.461889 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:52.462998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:52.463388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] PATH_INFO: `/v3/projects/0b10e28fddf44dcc9f11cdd65b178bdc` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:52.464598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b10e28fddf44dcc9f11cdd65b178bdc)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:52.467906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:52.519434 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b10e28fddf44dcc9f11cdd65b178bdc', 'request_id': 'req-3aed1c36-ea47-434a-b91f-299685ff2a27'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:52.525307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b10e28fddf44dcc9f11cdd65b178bdc', 'request_id': 'req-3aed1c36-ea47-434a-b91f-299685ff2a27'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:52.533051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b10e28fddf44dcc9f11cdd65b178bdc', 'request_id': 'req-3aed1c36-ea47-434a-b91f-299685ff2a27'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:52.545201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] The token cache is being invalidate because project 0b10e28fddf44dcc9f11cdd65b178bdc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:52.545485 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:52.546226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:52.547907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 976/1976] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:52 2022] DELETE /identity/v3/projects/0b10e28fddf44dcc9f11cdd65b178bdc => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:52.552173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d771250a-074a-4fcd-bd0a-70d89e4e8358 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:52.562398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d771250a-074a-4fcd-bd0a-70d89e4e8358 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:52.583489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:52.626510 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3839d4-43c1-4805-b25a-bc785e7b7aac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:52.627304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3839d4-43c1-4805-b25a-bc785e7b7aac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:52.627533 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3839d4-43c1-4805-b25a-bc785e7b7aac admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:52.627888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3839d4-43c1-4805-b25a-bc785e7b7aac admin admin] PATH_INFO: `/v3/users/e439ef3b1e7d47cca73cdfe67e8bb253` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:52.629281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3839d4-43c1-4805-b25a-bc785e7b7aac admin admin] RBAC: Authorizing `identity:delete_user(user_id=e439ef3b1e7d47cca73cdfe67e8bb253)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:52.631432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3839d4-43c1-4805-b25a-bc785e7b7aac admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:52.688213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8f3839d4-43c1-4805-b25a-bc785e7b7aac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e439ef3b1e7d47cca73cdfe67e8bb253'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:52.689067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8f3839d4-43c1-4805-b25a-bc785e7b7aac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e439ef3b1e7d47cca73cdfe67e8bb253'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:52.696457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8f3839d4-43c1-4805-b25a-bc785e7b7aac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e439ef3b1e7d47cca73cdfe67e8bb253'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:52.697165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8f3839d4-43c1-4805-b25a-bc785e7b7aac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e439ef3b1e7d47cca73cdfe67e8bb253'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:52.712757 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8f3839d4-43c1-4805-b25a-bc785e7b7aac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e439ef3b1e7d47cca73cdfe67e8bb253'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:52.733010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1001/1977] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:52 2022] DELETE /identity/v3/users/e439ef3b1e7d47cca73cdfe67e8bb253 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:53.436983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:53.438528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:53.455024 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aed1c36-ea47-434a-b91f-299685ff2a27 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:53.468538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:53.517789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:53.518411 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:53.518688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:53.518959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf admin admin] PATH_INFO: `/v3/projects/ec2219d8b0b34381a22658dfda92b8e1` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:53.520052 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=ec2219d8b0b34381a22658dfda92b8e1)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:53.521986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:53.565745 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ec2219d8b0b34381a22658dfda92b8e1', 'request_id': 'req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:53.570998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ec2219d8b0b34381a22658dfda92b8e1', 'request_id': 'req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:53.578546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ec2219d8b0b34381a22658dfda92b8e1', 'request_id': 'req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:53.589128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf admin admin] The token cache is being invalidate because project ec2219d8b0b34381a22658dfda92b8e1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:53.589412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:53.590025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ab73ef88-c9e6-4c80-b9d7-97484e4663cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:53.592116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 977/1978] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:53 2022] DELETE /identity/v3/projects/ec2219d8b0b34381a22658dfda92b8e1 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:53.597405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5574512b-fe74-42ff-9ed8-335233a09afa None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:53.597877 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5574512b-fe74-42ff-9ed8-335233a09afa None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:53.597990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5574512b-fe74-42ff-9ed8-335233a09afa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:53.617538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-5574512b-fe74-42ff-9ed8-335233a09afa None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:53.618786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5574512b-fe74-42ff-9ed8-335233a09afa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:53.678672 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1002/1979] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:53.686336 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa4cd6e-7896-4fc1-92fa-9014478695a5 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:53.686552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa4cd6e-7896-4fc1-92fa-9014478695a5 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:53.686828 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa4cd6e-7896-4fc1-92fa-9014478695a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:53.706919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-2fa4cd6e-7896-4fc1-92fa-9014478695a5 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:53.708300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2fa4cd6e-7896-4fc1-92fa-9014478695a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:53.716824 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 978/1980] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:53.721398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5574512b-fe74-42ff-9ed8-335233a09afa None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:53.730756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5574512b-fe74-42ff-9ed8-335233a09afa None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:53.735315 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc5f10f-9b63-4d4c-90a4-7b6a227f43aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:53.735911 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc5f10f-9b63-4d4c-90a4-7b6a227f43aa admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:53.736187 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc5f10f-9b63-4d4c-90a4-7b6a227f43aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:53.736503 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc5f10f-9b63-4d4c-90a4-7b6a227f43aa admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:53.737149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc5f10f-9b63-4d4c-90a4-7b6a227f43aa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:15:53.737450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc5f10f-9b63-4d4c-90a4-7b6a227f43aa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:53.739010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc5f10f-9b63-4d4c-90a4-7b6a227f43aa admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:53.754713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1003/1981] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:15:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:15:53.755906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa4cd6e-7896-4fc1-92fa-9014478695a5 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:53.765992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa4cd6e-7896-4fc1-92fa-9014478695a5 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:53.771554 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2919e4-ad2f-45cf-a1f0-07139ad35ed5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:53.772214 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2919e4-ad2f-45cf-a1f0-07139ad35ed5 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:53.772590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2919e4-ad2f-45cf-a1f0-07139ad35ed5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:53.772939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2919e4-ad2f-45cf-a1f0-07139ad35ed5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:53.773630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2919e4-ad2f-45cf-a1f0-07139ad35ed5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:53.775587 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2919e4-ad2f-45cf-a1f0-07139ad35ed5 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:53.802072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 979/1982] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:53 2022] POST /identity/v3/projects => generated 383 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:53.806351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc5f10f-9b63-4d4c-90a4-7b6a227f43aa admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:53.817360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc5f10f-9b63-4d4c-90a4-7b6a227f43aa admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:53.846406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:53.876007 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a04100-320f-430c-ba5c-18402c6e5a52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:53.876655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a04100-320f-430c-ba5c-18402c6e5a52 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:53.876934 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a04100-320f-430c-ba5c-18402c6e5a52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:53.877205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a04100-320f-430c-ba5c-18402c6e5a52 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:53.877837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a04100-320f-430c-ba5c-18402c6e5a52 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:53.880143 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a04100-320f-430c-ba5c-18402c6e5a52 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:53.901007 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1004/1983] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:53 2022] POST /identity/v3/users => generated 292 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:53.905252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2919e4-ad2f-45cf-a1f0-07139ad35ed5 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:53.915533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2919e4-ad2f-45cf-a1f0-07139ad35ed5 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:53.920348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9645139-0a88-4f5e-be01-84360c5e0b7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:53.920944 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9645139-0a88-4f5e-be01-84360c5e0b7a admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:53.921252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9645139-0a88-4f5e-be01-84360c5e0b7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:53.921520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9645139-0a88-4f5e-be01-84360c5e0b7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:53.922186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9645139-0a88-4f5e-be01-84360c5e0b7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:53.923767 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9645139-0a88-4f5e-be01-84360c5e0b7a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:53.934019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 980/1984] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:53 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:53.938396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a04100-320f-430c-ba5c-18402c6e5a52 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:53.947830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a04100-320f-430c-ba5c-18402c6e5a52 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:53.952937 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ade089-a362-4c5a-a32b-a3232fe0d532 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:53.953517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ade089-a362-4c5a-a32b-a3232fe0d532 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:53.953795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ade089-a362-4c5a-a32b-a3232fe0d532 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:53.954067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ade089-a362-4c5a-a32b-a3232fe0d532 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:53.954702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ade089-a362-4c5a-a32b-a3232fe0d532 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:53.956534 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ade089-a362-4c5a-a32b-a3232fe0d532 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:53.966913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1005/1985] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:53 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:53.971246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9645139-0a88-4f5e-be01-84360c5e0b7a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:53.981725 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9645139-0a88-4f5e-be01-84360c5e0b7a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:53.984497 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ade089-a362-4c5a-a32b-a3232fe0d532 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:53.986485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-04ade089-a362-4c5a-a32b-a3232fe0d532 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:53.986862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8685cf53-d62a-4168-9a4e-aab0ea7b56bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:53.987527 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685cf53-d62a-4168-9a4e-aab0ea7b56bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:53.987857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685cf53-d62a-4168-9a4e-aab0ea7b56bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:53.988080 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685cf53-d62a-4168-9a4e-aab0ea7b56bf admin admin] PATH_INFO: `/v3/projects/5fe6d64bcd684732a59bc9f0978dae07/users/916d2c6780d44f6ea17370b279d50eab/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.001175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ade089-a362-4c5a-a32b-a3232fe0d532 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:54.005861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8685cf53-d62a-4168-9a4e-aab0ea7b56bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=5fe6d64bcd684732a59bc9f0978dae07, user_id=916d2c6780d44f6ea17370b279d50eab, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:54.008025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb5d666-eec1-4147-aa32-233592a7bd07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:54.008575 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8685cf53-d62a-4168-9a4e-aab0ea7b56bf admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:54.009141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb5d666-eec1-4147-aa32-233592a7bd07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:54.009692 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb5d666-eec1-4147-aa32-233592a7bd07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:54.010345 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb5d666-eec1-4147-aa32-233592a7bd07 admin admin] PATH_INFO: `/v3/users/eae40347e449436184b62c3dd33966f0` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.011865 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb5d666-eec1-4147-aa32-233592a7bd07 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eae40347e449436184b62c3dd33966f0)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:54.015104 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb5d666-eec1-4147-aa32-233592a7bd07 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:54.023093 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 981/1986] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:53 2022] PUT /identity/v3/projects/5fe6d64bcd684732a59bc9f0978dae07/users/916d2c6780d44f6ea17370b279d50eab/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:54.037775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f37750-bbf3-46fb-ab41-1be54e0fe1a5 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:54.038251 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f37750-bbf3-46fb-ab41-1be54e0fe1a5 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:54.038903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f37750-bbf3-46fb-ab41-1be54e0fe1a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.079437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-beb5d666-eec1-4147-aa32-233592a7bd07 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eae40347e449436184b62c3dd33966f0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:54.080199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-beb5d666-eec1-4147-aa32-233592a7bd07 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eae40347e449436184b62c3dd33966f0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:54.082080 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-c7f37750-bbf3-46fb-ab41-1be54e0fe1a5 None None] MFA Rules not processed for user `916d2c6780d44f6ea17370b279d50eab`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:54.083282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c7f37750-bbf3-46fb-ab41-1be54e0fe1a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:54.088144 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-beb5d666-eec1-4147-aa32-233592a7bd07 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eae40347e449436184b62c3dd33966f0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:54.088918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-beb5d666-eec1-4147-aa32-233592a7bd07 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eae40347e449436184b62c3dd33966f0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:54.103841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-beb5d666-eec1-4147-aa32-233592a7bd07 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eae40347e449436184b62c3dd33966f0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:54.116848 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:54.121882 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1006/1987] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:15:53 2022] DELETE /identity/v3/users/eae40347e449436184b62c3dd33966f0 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:15:54.126760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f523ab17-6481-4a62-9a65-5727eab8e9b6 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:54.127055 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f523ab17-6481-4a62-9a65-5727eab8e9b6 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:54.127336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f523ab17-6481-4a62-9a65-5727eab8e9b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.150630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-f523ab17-6481-4a62-9a65-5727eab8e9b6 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:54.152156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f523ab17-6481-4a62-9a65-5727eab8e9b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:54.185230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 982/1988] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:54 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:54.190652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f37750-bbf3-46fb-ab41-1be54e0fe1a5 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:54.191959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c7f37750-bbf3-46fb-ab41-1be54e0fe1a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:54.224092 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1007/1989] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:54.234888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f523ab17-6481-4a62-9a65-5727eab8e9b6 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:54.236317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f523ab17-6481-4a62-9a65-5727eab8e9b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:54.264344 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:54.268807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f37750-bbf3-46fb-ab41-1be54e0fe1a5 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:54.274609 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824a0b4f-d560-44c6-91e1-00020a2f5634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:54.275519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a0b4f-d560-44c6-91e1-00020a2f5634 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:54.275688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a0b4f-d560-44c6-91e1-00020a2f5634 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:54.275938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a0b4f-d560-44c6-91e1-00020a2f5634 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.276632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824a0b4f-d560-44c6-91e1-00020a2f5634 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:54.278634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824a0b4f-d560-44c6-91e1-00020a2f5634 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:54.300571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f523ab17-6481-4a62-9a65-5727eab8e9b6 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:54.304527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b906697-d094-448a-a909-63d3e3e150cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:54.305146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b906697-d094-448a-a909-63d3e3e150cc service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:54.305457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b906697-d094-448a-a909-63d3e3e150cc service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:54.305732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b906697-d094-448a-a909-63d3e3e150cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.308346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 983/1990] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:54 2022] POST /identity/v3/projects => generated 383 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:54.313279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824a0b4f-d560-44c6-91e1-00020a2f5634 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:54.317743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b906697-d094-448a-a909-63d3e3e150cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:54.319524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b906697-d094-448a-a909-63d3e3e150cc service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:54.330553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1008/1991] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:15:54.390663 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824a0b4f-d560-44c6-91e1-00020a2f5634 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:54.396397 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11339f4f-afa0-47c8-a484-25164df392e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:54.397099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11339f4f-afa0-47c8-a484-25164df392e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:54.397295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11339f4f-afa0-47c8-a484-25164df392e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:54.397566 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11339f4f-afa0-47c8-a484-25164df392e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.398231 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11339f4f-afa0-47c8-a484-25164df392e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:54.400511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11339f4f-afa0-47c8-a484-25164df392e9 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:54.423414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 984/1992] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:54 2022] POST /identity/v3/users => generated 292 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:54.427492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b906697-d094-448a-a909-63d3e3e150cc service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:54.438409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b906697-d094-448a-a909-63d3e3e150cc service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:54.444620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc067618-8bf9-48b6-a61e-631335277a34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:54.445353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc067618-8bf9-48b6-a61e-631335277a34 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:54.445699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc067618-8bf9-48b6-a61e-631335277a34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:54.446087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc067618-8bf9-48b6-a61e-631335277a34 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.446852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc067618-8bf9-48b6-a61e-631335277a34 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:54.448522 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc067618-8bf9-48b6-a61e-631335277a34 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:54.464570 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1009/1993] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:54 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:54.467725 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11339f4f-afa0-47c8-a484-25164df392e9 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:54.479623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11339f4f-afa0-47c8-a484-25164df392e9 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:54.488414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f75e08-ac95-457e-acdc-05c285c602a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:54.489653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f75e08-ac95-457e-acdc-05c285c602a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:54.490074 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f75e08-ac95-457e-acdc-05c285c602a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:54.490445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f75e08-ac95-457e-acdc-05c285c602a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.491209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f75e08-ac95-457e-acdc-05c285c602a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:54.493094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f75e08-ac95-457e-acdc-05c285c602a1 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:54.509895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 985/1994] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:54 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:54.514519 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc067618-8bf9-48b6-a61e-631335277a34 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:54.527129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc067618-8bf9-48b6-a61e-631335277a34 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:54.532583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96afa27e-1198-47b3-bb80-660146fcd063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:54.533422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96afa27e-1198-47b3-bb80-660146fcd063 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:54.533822 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96afa27e-1198-47b3-bb80-660146fcd063 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:54.534188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96afa27e-1198-47b3-bb80-660146fcd063 admin admin] PATH_INFO: `/v3/projects/00296957288c4ad69fc435897be406c2/users/5fbec0379fb84e8980820e7673f2ca4e/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.550577 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96afa27e-1198-47b3-bb80-660146fcd063 admin admin] RBAC: Authorizing `identity:create_grant(project_id=00296957288c4ad69fc435897be406c2, user_id=5fbec0379fb84e8980820e7673f2ca4e, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:54.553621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96afa27e-1198-47b3-bb80-660146fcd063 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:54.571014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1010/1995] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:54 2022] PUT /identity/v3/projects/00296957288c4ad69fc435897be406c2/users/5fbec0379fb84e8980820e7673f2ca4e/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:54.581426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df7d800-72e5-452f-bfed-2a893fe69f46 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:54.581884 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df7d800-72e5-452f-bfed-2a893fe69f46 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:54.582208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df7d800-72e5-452f-bfed-2a893fe69f46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.616156 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-8df7d800-72e5-452f-bfed-2a893fe69f46 None None] MFA Rules not processed for user `5fbec0379fb84e8980820e7673f2ca4e`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:54.616156 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8df7d800-72e5-452f-bfed-2a893fe69f46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:54.648715 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:54.688255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 986/1996] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:54 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:54.692873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96afa27e-1198-47b3-bb80-660146fcd063 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:54.744327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df7d800-72e5-452f-bfed-2a893fe69f46 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:54.746615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8df7d800-72e5-452f-bfed-2a893fe69f46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:54.762450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96afa27e-1198-47b3-bb80-660146fcd063 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:54.767042 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5b4b60-f376-4b41-8b1d-67157bc10ba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:54.767659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5b4b60-f376-4b41-8b1d-67157bc10ba3 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:54.767935 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5b4b60-f376-4b41-8b1d-67157bc10ba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:54.768208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5b4b60-f376-4b41-8b1d-67157bc10ba3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.768894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5b4b60-f376-4b41-8b1d-67157bc10ba3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:54.770838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5b4b60-f376-4b41-8b1d-67157bc10ba3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:54.810070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1011/1997] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:54 2022] POST /identity/v3/projects => generated 383 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:54.814415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5b4b60-f376-4b41-8b1d-67157bc10ba3 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:54.845534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df7d800-72e5-452f-bfed-2a893fe69f46 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:54.884373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5b4b60-f376-4b41-8b1d-67157bc10ba3 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:54.889663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b420ccb1-e7cd-40de-9a52-e8317e9439c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:54.890277 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b420ccb1-e7cd-40de-9a52-e8317e9439c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:54.890558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b420ccb1-e7cd-40de-9a52-e8317e9439c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:54.890832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b420ccb1-e7cd-40de-9a52-e8317e9439c4 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.891466 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b420ccb1-e7cd-40de-9a52-e8317e9439c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:54.893564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b420ccb1-e7cd-40de-9a52-e8317e9439c4 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:54.913425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf1985a-f776-482c-8010-1a634dbfc4fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:54.913693 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf1985a-f776-482c-8010-1a634dbfc4fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:54.914027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf1985a-f776-482c-8010-1a634dbfc4fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:54.914314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf1985a-f776-482c-8010-1a634dbfc4fc admin admin] PATH_INFO: `/v3/projects/7cde07ab829342a5a57acc65f455a1af` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.915593 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf1985a-f776-482c-8010-1a634dbfc4fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=7cde07ab829342a5a57acc65f455a1af)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:54.917777 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf1985a-f776-482c-8010-1a634dbfc4fc admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:54.932334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1012/1998] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:54 2022] POST /identity/v3/users => generated 298 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:15:54.936010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b420ccb1-e7cd-40de-9a52-e8317e9439c4 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:54.946679 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:54.953873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b420ccb1-e7cd-40de-9a52-e8317e9439c4 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:54.962759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8cf4ec-4fc9-4d1c-ad81-f00cde77448f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:54.964062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8cf4ec-4fc9-4d1c-ad81-f00cde77448f admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:54.964456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8cf4ec-4fc9-4d1c-ad81-f00cde77448f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:54.964735 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8cf4ec-4fc9-4d1c-ad81-f00cde77448f admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:54.965400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8cf4ec-4fc9-4d1c-ad81-f00cde77448f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:54.967999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8cf4ec-4fc9-4d1c-ad81-f00cde77448f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:54.980543 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5bf1985a-f776-482c-8010-1a634dbfc4fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7cde07ab829342a5a57acc65f455a1af', 'request_id': 'req-5bf1985a-f776-482c-8010-1a634dbfc4fc'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:54.983828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1013/1999] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:54 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:15:54.987888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8cf4ec-4fc9-4d1c-ad81-f00cde77448f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:54.993187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5bf1985a-f776-482c-8010-1a634dbfc4fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7cde07ab829342a5a57acc65f455a1af', 'request_id': 'req-5bf1985a-f776-482c-8010-1a634dbfc4fc'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:54.999548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8cf4ec-4fc9-4d1c-ad81-f00cde77448f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:55.005544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5bf1985a-f776-482c-8010-1a634dbfc4fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7cde07ab829342a5a57acc65f455a1af', 'request_id': 'req-5bf1985a-f776-482c-8010-1a634dbfc4fc'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:55.026473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5bf1985a-f776-482c-8010-1a634dbfc4fc admin admin] The token cache is being invalidate because project 7cde07ab829342a5a57acc65f455a1af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:15:55.026819 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5bf1985a-f776-482c-8010-1a634dbfc4fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:55.028184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5bf1985a-f776-482c-8010-1a634dbfc4fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:15:55.030287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 987/2000] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:15:54 2022] DELETE /identity/v3/projects/7cde07ab829342a5a57acc65f455a1af => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:55.035427 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f25572-5496-4d06-b443-738a2fec08ec None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.035713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f25572-5496-4d06-b443-738a2fec08ec None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.035989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f25572-5496-4d06-b443-738a2fec08ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.044398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:55.057518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-e4f25572-5496-4d06-b443-738a2fec08ec None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:55.059457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e4f25572-5496-4d06-b443-738a2fec08ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:55.079202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9930971c-a53e-4e35-a16b-0b79fcf415bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:55.080441 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9930971c-a53e-4e35-a16b-0b79fcf415bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.080796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9930971c-a53e-4e35-a16b-0b79fcf415bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.081243 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9930971c-a53e-4e35-a16b-0b79fcf415bb admin admin] PATH_INFO: `/v3/projects/6ba00ace42f14b259c0c308c321669a2/users/58dce294837f4eb9a72e0623d4692ab8/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.105113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9930971c-a53e-4e35-a16b-0b79fcf415bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ba00ace42f14b259c0c308c321669a2, user_id=58dce294837f4eb9a72e0623d4692ab8, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:55.108359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9930971c-a53e-4e35-a16b-0b79fcf415bb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:55.126391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 988/2001] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:55.131399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1014/2002] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:54 2022] PUT /identity/v3/projects/6ba00ace42f14b259c0c308c321669a2/users/58dce294837f4eb9a72e0623d4692ab8/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:15:55.133694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f25572-5496-4d06-b443-738a2fec08ec None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:55.135053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6505a07-13c3-4147-bcb0-2c47fcf3b08e None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.135412 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6505a07-13c3-4147-bcb0-2c47fcf3b08e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.135786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6505a07-13c3-4147-bcb0-2c47fcf3b08e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.135894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e4f25572-5496-4d06-b443-738a2fec08ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:55.149781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f25572-5496-4d06-b443-738a2fec08ec None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:55.158295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-d6505a07-13c3-4147-bcb0-2c47fcf3b08e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:55.159694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d6505a07-13c3-4147-bcb0-2c47fcf3b08e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:55.225624 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf6c6bb-3530-40cb-9a52-2784dc04738e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:55.226425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf6c6bb-3530-40cb-9a52-2784dc04738e admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.226774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf6c6bb-3530-40cb-9a52-2784dc04738e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.227202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf6c6bb-3530-40cb-9a52-2784dc04738e admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.227931 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf6c6bb-3530-40cb-9a52-2784dc04738e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:55.229907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf6c6bb-3530-40cb-9a52-2784dc04738e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:55.241658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 989/2003] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:55 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:15:55.245737 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf6c6bb-3530-40cb-9a52-2784dc04738e admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:55.256542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf6c6bb-3530-40cb-9a52-2784dc04738e admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:55.261715 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1015/2004] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:55.261869 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517b1a15-0d10-4435-bf12-b77da87aecc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:55.263244 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517b1a15-0d10-4435-bf12-b77da87aecc6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.263630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517b1a15-0d10-4435-bf12-b77da87aecc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.264054 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517b1a15-0d10-4435-bf12-b77da87aecc6 admin admin] PATH_INFO: `/v3/projects/6ba00ace42f14b259c0c308c321669a2/users/58dce294837f4eb9a72e0623d4692ab8/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.266286 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517b1a15-0d10-4435-bf12-b77da87aecc6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ba00ace42f14b259c0c308c321669a2, user_id=58dce294837f4eb9a72e0623d4692ab8, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:55.266496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6505a07-13c3-4147-bcb0-2c47fcf3b08e None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:55.269204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517b1a15-0d10-4435-bf12-b77da87aecc6 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:55.277066 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6505a07-13c3-4147-bcb0-2c47fcf3b08e None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:55.283445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 990/2005] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:55 2022] PUT /identity/v3/projects/6ba00ace42f14b259c0c308c321669a2/users/58dce294837f4eb9a72e0623d4692ab8/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:15:55.293400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c193139-026e-4501-b26f-75ec05faf8f3 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.293759 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c193139-026e-4501-b26f-75ec05faf8f3 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.294038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c193139-026e-4501-b26f-75ec05faf8f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.326237 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-0c193139-026e-4501-b26f-75ec05faf8f3 None None] MFA Rules not processed for user `58dce294837f4eb9a72e0623d4692ab8`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:55.327468 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0c193139-026e-4501-b26f-75ec05faf8f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:55.350159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edced41-df4e-4afe-a554-718c5af0dfe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:55.351141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edced41-df4e-4afe-a554-718c5af0dfe2 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.351493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edced41-df4e-4afe-a554-718c5af0dfe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.351764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edced41-df4e-4afe-a554-718c5af0dfe2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.352438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edced41-df4e-4afe-a554-718c5af0dfe2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:15:55.352742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edced41-df4e-4afe-a554-718c5af0dfe2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:55.354310 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edced41-df4e-4afe-a554-718c5af0dfe2 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:55.378489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1016/2006] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:15:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:15:55.383569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edced41-df4e-4afe-a554-718c5af0dfe2 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:55.395051 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edced41-df4e-4afe-a554-718c5af0dfe2 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:55.399688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0857039f-8884-4576-82ea-3c223b4d8e6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:55.400279 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0857039f-8884-4576-82ea-3c223b4d8e6a admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.400625 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0857039f-8884-4576-82ea-3c223b4d8e6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.401016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0857039f-8884-4576-82ea-3c223b4d8e6a admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.401648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0857039f-8884-4576-82ea-3c223b4d8e6a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:55.403630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0857039f-8884-4576-82ea-3c223b4d8e6a admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:55.413911 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:55.415301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 991/2007] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:55 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:15:55.427520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c193139-026e-4501-b26f-75ec05faf8f3 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:55.429260 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0c193139-026e-4501-b26f-75ec05faf8f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:55.435940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1017/2008] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:15:55 2022] POST /identity/v3/projects => generated 399 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:15:55.440214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0857039f-8884-4576-82ea-3c223b4d8e6a admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:55.443428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:55.498795 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c193139-026e-4501-b26f-75ec05faf8f3 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:55.517071 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0857039f-8884-4576-82ea-3c223b4d8e6a admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:55.522338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948520dc-cbed-472f-ba5a-e8a287b3e9c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:55.522947 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948520dc-cbed-472f-ba5a-e8a287b3e9c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.523570 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948520dc-cbed-472f-ba5a-e8a287b3e9c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.523695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948520dc-cbed-472f-ba5a-e8a287b3e9c0 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.524288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948520dc-cbed-472f-ba5a-e8a287b3e9c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:55.526235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948520dc-cbed-472f-ba5a-e8a287b3e9c0 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:55.553146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1018/2009] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:15:55 2022] POST /identity/v3/users => generated 300 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:15:55.554437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948520dc-cbed-472f-ba5a-e8a287b3e9c0 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:55.562418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407512d2-9373-4e46-9280-99c4bf59f864 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:55.563127 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407512d2-9373-4e46-9280-99c4bf59f864 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.563480 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407512d2-9373-4e46-9280-99c4bf59f864 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.563832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407512d2-9373-4e46-9280-99c4bf59f864 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.565325 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948520dc-cbed-472f-ba5a-e8a287b3e9c0 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:55.565776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-407512d2-9373-4e46-9280-99c4bf59f864 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:55.569842 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd6a48d-16b3-45d8-9f4b-9151e516c894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:55.570426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd6a48d-16b3-45d8-9f4b-9151e516c894 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.570705 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd6a48d-16b3-45d8-9f4b-9151e516c894 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.570976 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd6a48d-16b3-45d8-9f4b-9151e516c894 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.571617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd6a48d-16b3-45d8-9f4b-9151e516c894 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:55.573258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd6a48d-16b3-45d8-9f4b-9151e516c894 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:55.586546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1019/2010] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:55 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:15:55.587735 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd6a48d-16b3-45d8-9f4b-9151e516c894 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:55.598394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd6a48d-16b3-45d8-9f4b-9151e516c894 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:55.602941 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7ea4b5-1b1f-4de1-9849-bd0355a87fa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:55.603511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7ea4b5-1b1f-4de1-9849-bd0355a87fa7 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.603790 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7ea4b5-1b1f-4de1-9849-bd0355a87fa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.604091 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7ea4b5-1b1f-4de1-9849-bd0355a87fa7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.604874 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7ea4b5-1b1f-4de1-9849-bd0355a87fa7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:55.607021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7ea4b5-1b1f-4de1-9849-bd0355a87fa7 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:55.620683 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1020/2011] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:15:55 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:15:55.622081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7ea4b5-1b1f-4de1-9849-bd0355a87fa7 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:55.628446 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407512d2-9373-4e46-9280-99c4bf59f864 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:55.629900 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407512d2-9373-4e46-9280-99c4bf59f864 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:55.631535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7ea4b5-1b1f-4de1-9849-bd0355a87fa7 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:55.637319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3861f63b-6348-4815-a277-807efd7e251c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:55.638491 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3861f63b-6348-4815-a277-807efd7e251c admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.638772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3861f63b-6348-4815-a277-807efd7e251c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.639051 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3861f63b-6348-4815-a277-807efd7e251c admin admin] PATH_INFO: `/v3/projects/a873fadb519f4caf9592ea21ff158531/users/57b1f68b190a4ca68a0cf690a3919873/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.640253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 992/2012] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:55 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:15:55.659648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3861f63b-6348-4815-a277-807efd7e251c admin admin] RBAC: Authorizing `identity:create_grant(project_id=a873fadb519f4caf9592ea21ff158531, user_id=57b1f68b190a4ca68a0cf690a3919873, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:55.662669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3861f63b-6348-4815-a277-807efd7e251c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:55.680485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1021/2013] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:15:55 2022] PUT /identity/v3/projects/a873fadb519f4caf9592ea21ff158531/users/57b1f68b190a4ca68a0cf690a3919873/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:15:55.691153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ef14d-543f-4c59-a013-7b9d021779d0 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.692036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ef14d-543f-4c59-a013-7b9d021779d0 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.692510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ef14d-543f-4c59-a013-7b9d021779d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.731095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-327ef14d-543f-4c59-a013-7b9d021779d0 None None] MFA Rules not processed for user `57b1f68b190a4ca68a0cf690a3919873`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:15:55.733185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-327ef14d-543f-4c59-a013-7b9d021779d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:55.833106 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 993/2014] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:15:55 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:15:55.849668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3861f63b-6348-4815-a277-807efd7e251c admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:55.851518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-3861f63b-6348-4815-a277-807efd7e251c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:55.923964 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3861f63b-6348-4815-a277-807efd7e251c admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:55.928634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ee3bbf-50bd-43b9-8001-d88636abaa54 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:55.929262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ee3bbf-50bd-43b9-8001-d88636abaa54 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:55.929566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ee3bbf-50bd-43b9-8001-d88636abaa54 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:55.929811 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ee3bbf-50bd-43b9-8001-d88636abaa54 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:55.939472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ee3bbf-50bd-43b9-8001-d88636abaa54 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:55.941325 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ee3bbf-50bd-43b9-8001-d88636abaa54 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:55.958474 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1022/2015] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:55 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:15:59.392975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327ef14d-543f-4c59-a013-7b9d021779d0 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:15:59.405764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327ef14d-543f-4c59-a013-7b9d021779d0 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:15:59.424728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98deb4d-dae1-445a-a2c9-89e62661f157 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:15:59.425410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98deb4d-dae1-445a-a2c9-89e62661f157 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:15:59.425868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98deb4d-dae1-445a-a2c9-89e62661f157 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:15:59.426292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98deb4d-dae1-445a-a2c9-89e62661f157 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:15:59.428198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d98deb4d-dae1-445a-a2c9-89e62661f157 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:15:59.438633 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:15:59.509107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98deb4d-dae1-445a-a2c9-89e62661f157 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:15:59.510490 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98deb4d-dae1-445a-a2c9-89e62661f157 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:15:59.519659 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 994/2016] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:15:59 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:16:00.230382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ee3bbf-50bd-43b9-8001-d88636abaa54 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:00.239402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ee3bbf-50bd-43b9-8001-d88636abaa54 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:00.243329 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ec5065-785a-4f78-9e79-1971f4cbe684 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:00.243924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ec5065-785a-4f78-9e79-1971f4cbe684 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:00.244199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ec5065-785a-4f78-9e79-1971f4cbe684 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:00.244498 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ec5065-785a-4f78-9e79-1971f4cbe684 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:00.279174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:00.308197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ec5065-785a-4f78-9e79-1971f4cbe684 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:00.309884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ec5065-785a-4f78-9e79-1971f4cbe684 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:00.319620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1023/2017] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:16:00 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:16:06.372597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98deb4d-dae1-445a-a2c9-89e62661f157 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:06.374069 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d98deb4d-dae1-445a-a2c9-89e62661f157 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:06.439645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98deb4d-dae1-445a-a2c9-89e62661f157 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:06.444243 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba5e9d8-e8b2-4c97-8e39-a0aaa7ecfec7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:06.444934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba5e9d8-e8b2-4c97-8e39-a0aaa7ecfec7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:06.445218 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba5e9d8-e8b2-4c97-8e39-a0aaa7ecfec7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:06.445493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba5e9d8-e8b2-4c97-8e39-a0aaa7ecfec7 admin admin] PATH_INFO: `/v3/users/9bb923e0d3fb49cfb58b8b4329fedae4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:06.446624 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba5e9d8-e8b2-4c97-8e39-a0aaa7ecfec7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9bb923e0d3fb49cfb58b8b4329fedae4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:06.448575 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba5e9d8-e8b2-4c97-8e39-a0aaa7ecfec7 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:06.499505 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1ba5e9d8-e8b2-4c97-8e39-a0aaa7ecfec7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9bb923e0d3fb49cfb58b8b4329fedae4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:06.500193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1ba5e9d8-e8b2-4c97-8e39-a0aaa7ecfec7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9bb923e0d3fb49cfb58b8b4329fedae4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:06.500850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1ba5e9d8-e8b2-4c97-8e39-a0aaa7ecfec7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9bb923e0d3fb49cfb58b8b4329fedae4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:06.508334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1ba5e9d8-e8b2-4c97-8e39-a0aaa7ecfec7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9bb923e0d3fb49cfb58b8b4329fedae4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:06.521306 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1ba5e9d8-e8b2-4c97-8e39-a0aaa7ecfec7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9bb923e0d3fb49cfb58b8b4329fedae4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:06.538367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 995/2018] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:16:06 2022] DELETE /identity/v3/users/9bb923e0d3fb49cfb58b8b4329fedae4 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:06.543096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c29a62e-3603-40ec-ae40-9ae8ba106c10 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:06.544189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c29a62e-3603-40ec-ae40-9ae8ba106c10 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:06.544327 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c29a62e-3603-40ec-ae40-9ae8ba106c10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:06.563951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-7c29a62e-3603-40ec-ae40-9ae8ba106c10 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:06.565226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7c29a62e-3603-40ec-ae40-9ae8ba106c10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:06.629077 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1024/2019] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:16:06.639447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba5e9d8-e8b2-4c97-8e39-a0aaa7ecfec7 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:06.640945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1ba5e9d8-e8b2-4c97-8e39-a0aaa7ecfec7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:06.673595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:06.702884 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba5e9d8-e8b2-4c97-8e39-a0aaa7ecfec7 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:06.706614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2348d995-c388-4623-a79e-7be1070ff72a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:06.707228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2348d995-c388-4623-a79e-7be1070ff72a service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:06.707510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2348d995-c388-4623-a79e-7be1070ff72a service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:06.709060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2348d995-c388-4623-a79e-7be1070ff72a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:06.717676 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2348d995-c388-4623-a79e-7be1070ff72a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:06.719130 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2348d995-c388-4623-a79e-7be1070ff72a service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:06.728913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 996/2020] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:16:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:16:07.105015 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c29a62e-3603-40ec-ae40-9ae8ba106c10 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:07.107680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7c29a62e-3603-40ec-ae40-9ae8ba106c10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:07.120279 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c29a62e-3603-40ec-ae40-9ae8ba106c10 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:07.128496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:07.128496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:07.136529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:07.138289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] PATH_INFO: `/v3/projects/258d9e33e4d54f9894f1d54c53e3b4ba` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:07.140551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] RBAC: Authorizing `identity:delete_project(project_id=258d9e33e4d54f9894f1d54c53e3b4ba)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:07.142784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:07.205793 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '258d9e33e4d54f9894f1d54c53e3b4ba', 'request_id': 'req-90416f13-b824-413d-a897-1535a6df2660'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:07.227382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '258d9e33e4d54f9894f1d54c53e3b4ba', 'request_id': 'req-90416f13-b824-413d-a897-1535a6df2660'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:07.256113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '258d9e33e4d54f9894f1d54c53e3b4ba', 'request_id': 'req-90416f13-b824-413d-a897-1535a6df2660'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:07.279509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] The token cache is being invalidate because project 258d9e33e4d54f9894f1d54c53e3b4ba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:16:07.280483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:07.282087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:07.293025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1025/2021] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:16:07 2022] DELETE /identity/v3/projects/258d9e33e4d54f9894f1d54c53e3b4ba => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:07.303318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fea30a0-ab07-40ab-9def-8e88f89a3e5c None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:07.305158 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fea30a0-ab07-40ab-9def-8e88f89a3e5c None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:07.305447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fea30a0-ab07-40ab-9def-8e88f89a3e5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:07.334017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:07.335458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:07.357643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-0fea30a0-ab07-40ab-9def-8e88f89a3e5c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:07.358937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0fea30a0-ab07-40ab-9def-8e88f89a3e5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:07.407797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:07.438505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90416f13-b824-413d-a897-1535a6df2660 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:07.462265 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc91694e-1e29-4a67-b30d-9ada38123667 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:07.463054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc91694e-1e29-4a67-b30d-9ada38123667 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:07.463434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc91694e-1e29-4a67-b30d-9ada38123667 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:07.463805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc91694e-1e29-4a67-b30d-9ada38123667 admin admin] PATH_INFO: `/v3/users/57b1f68b190a4ca68a0cf690a3919873` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:07.465830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc91694e-1e29-4a67-b30d-9ada38123667 admin admin] RBAC: Authorizing `identity:delete_user(user_id=57b1f68b190a4ca68a0cf690a3919873)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:07.469679 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc91694e-1e29-4a67-b30d-9ada38123667 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:07.512680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 997/2022] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:07.523205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172fd960-b95c-4364-bed6-c777d1812907 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:07.523652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172fd960-b95c-4364-bed6-c777d1812907 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:07.523964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172fd960-b95c-4364-bed6-c777d1812907 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:07.555445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fc91694e-1e29-4a67-b30d-9ada38123667 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '57b1f68b190a4ca68a0cf690a3919873'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:07.556143 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fc91694e-1e29-4a67-b30d-9ada38123667 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '57b1f68b190a4ca68a0cf690a3919873'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:07.562808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-172fd960-b95c-4364-bed6-c777d1812907 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:07.565229 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fc91694e-1e29-4a67-b30d-9ada38123667 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '57b1f68b190a4ca68a0cf690a3919873'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:07.566507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fc91694e-1e29-4a67-b30d-9ada38123667 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '57b1f68b190a4ca68a0cf690a3919873'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:07.567373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-172fd960-b95c-4364-bed6-c777d1812907 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:07.585760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fc91694e-1e29-4a67-b30d-9ada38123667 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '57b1f68b190a4ca68a0cf690a3919873'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:07.589109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 998/2023] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:07.591477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172fd960-b95c-4364-bed6-c777d1812907 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:07.604629 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172fd960-b95c-4364-bed6-c777d1812907 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:07.610635 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1026/2024] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:16:07 2022] DELETE /identity/v3/users/57b1f68b190a4ca68a0cf690a3919873 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:07.614908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e526736d-44b5-4a3b-9049-21fe5bf66feb None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:07.615449 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e526736d-44b5-4a3b-9049-21fe5bf66feb None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:07.615857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5792df52-2157-4060-bec2-db8cd58bd423 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:07.616044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e526736d-44b5-4a3b-9049-21fe5bf66feb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:07.616655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5792df52-2157-4060-bec2-db8cd58bd423 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:07.616991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5792df52-2157-4060-bec2-db8cd58bd423 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:07.617327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5792df52-2157-4060-bec2-db8cd58bd423 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:07.618042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5792df52-2157-4060-bec2-db8cd58bd423 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:16:07.618405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5792df52-2157-4060-bec2-db8cd58bd423 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:07.620297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5792df52-2157-4060-bec2-db8cd58bd423 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:07.634110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 999/2025] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:16:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:16:07.639042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5792df52-2157-4060-bec2-db8cd58bd423 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:07.639289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-e526736d-44b5-4a3b-9049-21fe5bf66feb None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:07.640543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e526736d-44b5-4a3b-9049-21fe5bf66feb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:07.649295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5792df52-2157-4060-bec2-db8cd58bd423 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:07.714458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1027/2026] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:16:07.725957 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e526736d-44b5-4a3b-9049-21fe5bf66feb None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:07.730906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e526736d-44b5-4a3b-9049-21fe5bf66feb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:07.742722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b159235f-c7f6-4dcc-8383-ce26951437f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:07.743597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b159235f-c7f6-4dcc-8383-ce26951437f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:07.744031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b159235f-c7f6-4dcc-8383-ce26951437f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:07.744647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b159235f-c7f6-4dcc-8383-ce26951437f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:07.745325 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b159235f-c7f6-4dcc-8383-ce26951437f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:07.748344 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b159235f-c7f6-4dcc-8383-ce26951437f1 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:07.787560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1000/2027] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:16:07 2022] POST /identity/v3/projects => generated 369 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:16:07.788635 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b159235f-c7f6-4dcc-8383-ce26951437f1 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:07.811968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e526736d-44b5-4a3b-9049-21fe5bf66feb None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:07.843338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:07.846601 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:07.870811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b159235f-c7f6-4dcc-8383-ce26951437f1 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:07.877388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c72c87c-9bc4-43e4-805f-21d14bd57408 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:07.877623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fa7c04-462a-4700-9880-9ab5e0a5c956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:07.878232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c72c87c-9bc4-43e4-805f-21d14bd57408 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:07.878375 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fa7c04-462a-4700-9880-9ab5e0a5c956 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:07.879086 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fa7c04-462a-4700-9880-9ab5e0a5c956 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:07.879235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c72c87c-9bc4-43e4-805f-21d14bd57408 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:07.879331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fa7c04-462a-4700-9880-9ab5e0a5c956 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:07.879418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c72c87c-9bc4-43e4-805f-21d14bd57408 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:07.879567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fa7c04-462a-4700-9880-9ab5e0a5c956 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:07.881625 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fa7c04-462a-4700-9880-9ab5e0a5c956 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:07.890049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c72c87c-9bc4-43e4-805f-21d14bd57408 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:07.892422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c72c87c-9bc4-43e4-805f-21d14bd57408 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:07.905328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1028/2028] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:16:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:16:07.914370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1001/2029] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:16:07 2022] POST /identity/v3/users => generated 285 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:16:07.915314 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c72c87c-9bc4-43e4-805f-21d14bd57408 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:07.926376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c72c87c-9bc4-43e4-805f-21d14bd57408 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:07.931191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb2a660-5780-4c2c-8fa0-cf9114ea9d89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:07.931815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb2a660-5780-4c2c-8fa0-cf9114ea9d89 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:07.932095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb2a660-5780-4c2c-8fa0-cf9114ea9d89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:07.932398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb2a660-5780-4c2c-8fa0-cf9114ea9d89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:07.933290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb2a660-5780-4c2c-8fa0-cf9114ea9d89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:07.934739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb2a660-5780-4c2c-8fa0-cf9114ea9d89 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:07.945443 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1029/2030] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:16:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:16:07.949241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fa7c04-462a-4700-9880-9ab5e0a5c956 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:07.962388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fa7c04-462a-4700-9880-9ab5e0a5c956 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:07.968800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac8a5d3-fe46-4fb7-8707-9b886246911f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:07.981973 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac8a5d3-fe46-4fb7-8707-9b886246911f admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:07.982321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac8a5d3-fe46-4fb7-8707-9b886246911f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:07.982740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac8a5d3-fe46-4fb7-8707-9b886246911f admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:07.983924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac8a5d3-fe46-4fb7-8707-9b886246911f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:07.985826 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac8a5d3-fe46-4fb7-8707-9b886246911f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:08.003138 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1002/2031] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:16:07 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:16:08.004069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb2a660-5780-4c2c-8fa0-cf9114ea9d89 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:08.015736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb2a660-5780-4c2c-8fa0-cf9114ea9d89 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:08.022260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b94ef2-6cfa-438b-8aaf-e7bf8e73d9b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:08.022922 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b94ef2-6cfa-438b-8aaf-e7bf8e73d9b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:08.023193 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b94ef2-6cfa-438b-8aaf-e7bf8e73d9b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:08.023597 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b94ef2-6cfa-438b-8aaf-e7bf8e73d9b3 admin admin] PATH_INFO: `/v3/projects/5f63c1760e3642d4b6a3f5c499832072/users/ca2c40f0872e4601b28e365f882a0adf/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:08.043214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b94ef2-6cfa-438b-8aaf-e7bf8e73d9b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f63c1760e3642d4b6a3f5c499832072, user_id=ca2c40f0872e4601b28e365f882a0adf, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:08.050803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b94ef2-6cfa-438b-8aaf-e7bf8e73d9b3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:08.100454 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1030/2032] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:16:08 2022] PUT /identity/v3/projects/5f63c1760e3642d4b6a3f5c499832072/users/ca2c40f0872e4601b28e365f882a0adf/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:16:08.111534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832fc4f8-bd79-481d-a885-6f1ef75f42fb None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:08.111937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832fc4f8-bd79-481d-a885-6f1ef75f42fb None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:08.112335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832fc4f8-bd79-481d-a885-6f1ef75f42fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:08.160927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-832fc4f8-bd79-481d-a885-6f1ef75f42fb None None] MFA Rules not processed for user `ca2c40f0872e4601b28e365f882a0adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:08.162859 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-832fc4f8-bd79-481d-a885-6f1ef75f42fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:08.245945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1003/2033] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:08 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:08.258646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b94ef2-6cfa-438b-8aaf-e7bf8e73d9b3 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:08.261256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d4b94ef2-6cfa-438b-8aaf-e7bf8e73d9b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:08.299990 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832fc4f8-bd79-481d-a885-6f1ef75f42fb None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:08.302174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-832fc4f8-bd79-481d-a885-6f1ef75f42fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:08.350721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b94ef2-6cfa-438b-8aaf-e7bf8e73d9b3 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:08.355081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398f062-3a45-4631-ba71-626eac0e7fc1 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '815504693bcc40b6aef06aacb2a9a35b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:08.356527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398f062-3a45-4631-ba71-626eac0e7fc1 service glance] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:08.357034 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398f062-3a45-4631-ba71-626eac0e7fc1 service glance] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:08.357505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398f062-3a45-4631-ba71-626eac0e7fc1 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:08.368918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398f062-3a45-4631-ba71-626eac0e7fc1 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:08.370876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398f062-3a45-4631-ba71-626eac0e7fc1 service glance] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:08.380592 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1031/2034] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:16:08 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:16:08.390497 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832fc4f8-bd79-481d-a885-6f1ef75f42fb None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:08.395294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2bd9fa-9534-4010-97e0-8e307984221a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:08.395932 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2bd9fa-9534-4010-97e0-8e307984221a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:08.396212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2bd9fa-9534-4010-97e0-8e307984221a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:08.396516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2bd9fa-9534-4010-97e0-8e307984221a admin admin] PATH_INFO: `/v3/projects/a873fadb519f4caf9592ea21ff158531` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:08.397683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2bd9fa-9534-4010-97e0-8e307984221a admin admin] RBAC: Authorizing `identity:delete_project(project_id=a873fadb519f4caf9592ea21ff158531)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:08.399727 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2bd9fa-9534-4010-97e0-8e307984221a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:08.451738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:08.453189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-eb2bd9fa-9534-4010-97e0-8e307984221a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a873fadb519f4caf9592ea21ff158531', 'request_id': 'req-eb2bd9fa-9534-4010-97e0-8e307984221a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:08.462955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-eb2bd9fa-9534-4010-97e0-8e307984221a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a873fadb519f4caf9592ea21ff158531', 'request_id': 'req-eb2bd9fa-9534-4010-97e0-8e307984221a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:08.498377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-eb2bd9fa-9534-4010-97e0-8e307984221a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a873fadb519f4caf9592ea21ff158531', 'request_id': 'req-eb2bd9fa-9534-4010-97e0-8e307984221a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:08.524820 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-eb2bd9fa-9534-4010-97e0-8e307984221a admin admin] The token cache is being invalidate because project a873fadb519f4caf9592ea21ff158531 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:16:08.525457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-eb2bd9fa-9534-4010-97e0-8e307984221a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:08.527799 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-eb2bd9fa-9534-4010-97e0-8e307984221a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:08.529971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1004/2035] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:16:08 2022] DELETE /identity/v3/projects/a873fadb519f4caf9592ea21ff158531 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:16:08.534876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dac22ac-54ae-4b49-9c4c-c2bd87dc8d04 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:08.535156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dac22ac-54ae-4b49-9c4c-c2bd87dc8d04 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:08.535424 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dac22ac-54ae-4b49-9c4c-c2bd87dc8d04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:08.594579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-7dac22ac-54ae-4b49-9c4c-c2bd87dc8d04 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:08.595724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7dac22ac-54ae-4b49-9c4c-c2bd87dc8d04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:08.753961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1032/2036] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:08.761928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40facf2d-d6a5-48dc-9b7e-8d31c54e449b None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:08.762516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40facf2d-d6a5-48dc-9b7e-8d31c54e449b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:08.763176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40facf2d-d6a5-48dc-9b7e-8d31c54e449b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:08.797296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dac22ac-54ae-4b49-9c4c-c2bd87dc8d04 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:08.799124 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7dac22ac-54ae-4b49-9c4c-c2bd87dc8d04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:08.822650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dac22ac-54ae-4b49-9c4c-c2bd87dc8d04 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:08.830442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04c4b95-9e50-4939-92d1-bbec99de489f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:08.832004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04c4b95-9e50-4939-92d1-bbec99de489f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:08.832505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04c4b95-9e50-4939-92d1-bbec99de489f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:08.832952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04c4b95-9e50-4939-92d1-bbec99de489f admin admin] PATH_INFO: `/v3/users/916d2c6780d44f6ea17370b279d50eab` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:08.834561 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04c4b95-9e50-4939-92d1-bbec99de489f admin admin] RBAC: Authorizing `identity:delete_user(user_id=916d2c6780d44f6ea17370b279d50eab)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:08.838498 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04c4b95-9e50-4939-92d1-bbec99de489f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:08.841032 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-40facf2d-d6a5-48dc-9b7e-8d31c54e449b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:08.842251 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-40facf2d-d6a5-48dc-9b7e-8d31c54e449b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:08.855292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1005/2037] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:08.861206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40facf2d-d6a5-48dc-9b7e-8d31c54e449b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:08.871970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40facf2d-d6a5-48dc-9b7e-8d31c54e449b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:08.877748 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f4e88c-085b-4563-8431-e154af134a87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:08.878614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f4e88c-085b-4563-8431-e154af134a87 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:08.878963 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f4e88c-085b-4563-8431-e154af134a87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:08.879637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f4e88c-085b-4563-8431-e154af134a87 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:08.882248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f4e88c-085b-4563-8431-e154af134a87 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:16:08.882633 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f4e88c-085b-4563-8431-e154af134a87 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:08.884384 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f4e88c-085b-4563-8431-e154af134a87 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:08.900421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1006/2038] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:16:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:16:08.904142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f4e88c-085b-4563-8431-e154af134a87 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:08.906764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:08.928350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f04c4b95-9e50-4939-92d1-bbec99de489f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '916d2c6780d44f6ea17370b279d50eab'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:08.929560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f04c4b95-9e50-4939-92d1-bbec99de489f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '916d2c6780d44f6ea17370b279d50eab'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:08.932962 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f4e88c-085b-4563-8431-e154af134a87 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:08.941041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef74b249-3ec3-428d-861b-0b0bfccec7fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:08.942120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef74b249-3ec3-428d-861b-0b0bfccec7fc admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:08.942120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef74b249-3ec3-428d-861b-0b0bfccec7fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:08.942323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef74b249-3ec3-428d-861b-0b0bfccec7fc admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:08.943029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef74b249-3ec3-428d-861b-0b0bfccec7fc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:08.946657 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef74b249-3ec3-428d-861b-0b0bfccec7fc admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:08.947002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f04c4b95-9e50-4939-92d1-bbec99de489f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '916d2c6780d44f6ea17370b279d50eab'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:08.948034 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f04c4b95-9e50-4939-92d1-bbec99de489f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '916d2c6780d44f6ea17370b279d50eab'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:08.964919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f04c4b95-9e50-4939-92d1-bbec99de489f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '916d2c6780d44f6ea17370b279d50eab'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:08.988366 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1033/2039] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:16:08 2022] DELETE /identity/v3/users/916d2c6780d44f6ea17370b279d50eab => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:16:08.988639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1007/2040] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:16:08 2022] POST /identity/v3/projects => generated 383 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:16:08.994212 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04c4b95-9e50-4939-92d1-bbec99de489f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:08.994212 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f04c4b95-9e50-4939-92d1-bbec99de489f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:08.995886 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabc8a6e-a8ea-4bbd-a1cc-7674a581ee84 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:08.996383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabc8a6e-a8ea-4bbd-a1cc-7674a581ee84 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:08.997523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabc8a6e-a8ea-4bbd-a1cc-7674a581ee84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:09.025775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-fabc8a6e-a8ea-4bbd-a1cc-7674a581ee84 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:09.028191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-fabc8a6e-a8ea-4bbd-a1cc-7674a581ee84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:09.101872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04c4b95-9e50-4939-92d1-bbec99de489f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:09.109022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50f08c2-7382-4b1c-8261-9d06e188c814 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:09.109889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50f08c2-7382-4b1c-8261-9d06e188c814 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:09.109889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50f08c2-7382-4b1c-8261-9d06e188c814 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:09.110165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50f08c2-7382-4b1c-8261-9d06e188c814 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:09.110912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50f08c2-7382-4b1c-8261-9d06e188c814 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:09.114396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50f08c2-7382-4b1c-8261-9d06e188c814 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:09.149544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1034/2041] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:16:08 2022] POST /identity/v3/users => generated 292 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:16:09.150717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50f08c2-7382-4b1c-8261-9d06e188c814 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:09.175248 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50f08c2-7382-4b1c-8261-9d06e188c814 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:09.185707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1008/2042] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:09.193657 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac556058-c42f-4134-bb1c-249be645eaa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:09.194481 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabc8a6e-a8ea-4bbd-a1cc-7674a581ee84 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:09.195053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac556058-c42f-4134-bb1c-249be645eaa5 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:09.195458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac556058-c42f-4134-bb1c-249be645eaa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:09.195939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac556058-c42f-4134-bb1c-249be645eaa5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:09.196250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-fabc8a6e-a8ea-4bbd-a1cc-7674a581ee84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:09.197016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac556058-c42f-4134-bb1c-249be645eaa5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:09.199515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac556058-c42f-4134-bb1c-249be645eaa5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:09.218301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1035/2043] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:16:09 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:16:09.221725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac556058-c42f-4134-bb1c-249be645eaa5 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:09.245341 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac556058-c42f-4134-bb1c-249be645eaa5 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:09.251431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b8bd17-fb1c-43f4-aca1-a9366820c87f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:09.252220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b8bd17-fb1c-43f4-aca1-a9366820c87f admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:09.252689 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b8bd17-fb1c-43f4-aca1-a9366820c87f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:09.253247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b8bd17-fb1c-43f4-aca1-a9366820c87f admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:09.254809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b8bd17-fb1c-43f4-aca1-a9366820c87f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:09.256823 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b8bd17-fb1c-43f4-aca1-a9366820c87f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:09.271838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1036/2044] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:16:09 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:16:09.276980 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b8bd17-fb1c-43f4-aca1-a9366820c87f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:09.302230 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b8bd17-fb1c-43f4-aca1-a9366820c87f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:09.309570 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b225ea-a3cf-44b5-91ea-a972f0c2b8ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:09.310486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b225ea-a3cf-44b5-91ea-a972f0c2b8ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:09.310763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b225ea-a3cf-44b5-91ea-a972f0c2b8ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:09.311217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b225ea-a3cf-44b5-91ea-a972f0c2b8ef admin admin] PATH_INFO: `/v3/projects/e271418048c1466e948bf84de04fc05f/users/4226c61349b440c3a502a7cbc592d883/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:09.328629 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabc8a6e-a8ea-4bbd-a1cc-7674a581ee84 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:09.332855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7c1331-3383-4bc0-ac29-06b932ed4d8d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:09.333468 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7c1331-3383-4bc0-ac29-06b932ed4d8d service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:09.333746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7c1331-3383-4bc0-ac29-06b932ed4d8d service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:09.334018 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7c1331-3383-4bc0-ac29-06b932ed4d8d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:09.336366 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b225ea-a3cf-44b5-91ea-a972f0c2b8ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=e271418048c1466e948bf84de04fc05f, user_id=4226c61349b440c3a502a7cbc592d883, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:09.339545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b225ea-a3cf-44b5-91ea-a972f0c2b8ef admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:09.350274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7c1331-3383-4bc0-ac29-06b932ed4d8d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:09.351949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7c1331-3383-4bc0-ac29-06b932ed4d8d service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:09.361786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1037/2045] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:16:09 2022] PUT /identity/v3/projects/e271418048c1466e948bf84de04fc05f/users/4226c61349b440c3a502a7cbc592d883/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:16:09.366989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1009/2046] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:16:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:16:09.367677 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736a241c-9f06-459b-8bcb-fb44308e951f None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:09.368183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736a241c-9f06-459b-8bcb-fb44308e951f None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:09.368603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736a241c-9f06-459b-8bcb-fb44308e951f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:09.390172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-736a241c-9f06-459b-8bcb-fb44308e951f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:09.391580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-736a241c-9f06-459b-8bcb-fb44308e951f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:09.449266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:09.470781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1038/2047] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:09.494381 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7c1331-3383-4bc0-ac29-06b932ed4d8d service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:09.500908 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3a7c1331-3383-4bc0-ac29-06b932ed4d8d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:09.525761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7c1331-3383-4bc0-ac29-06b932ed4d8d service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:09.576752 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=95,keystone:DELETE=3,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:09.596136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0adc91e8-2c3a-4fbd-8d00-412392f639d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:09.597057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adc91e8-2c3a-4fbd-8d00-412392f639d1 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:09.597579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adc91e8-2c3a-4fbd-8d00-412392f639d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:09.598149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adc91e8-2c3a-4fbd-8d00-412392f639d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:09.611057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0adc91e8-2c3a-4fbd-8d00-412392f639d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:09.615430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0adc91e8-2c3a-4fbd-8d00-412392f639d1 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:09.628494 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1010/2048] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:16:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:16:10.040631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736a241c-9f06-459b-8bcb-fb44308e951f None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:10.044184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-736a241c-9f06-459b-8bcb-fb44308e951f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:10.058350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736a241c-9f06-459b-8bcb-fb44308e951f None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:10.064557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:10.065565 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:10.066234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:10.066735 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] PATH_INFO: `/v3/projects/5fe6d64bcd684732a59bc9f0978dae07` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:10.068662 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5fe6d64bcd684732a59bc9f0978dae07)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:10.072127 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:10.151529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5fe6d64bcd684732a59bc9f0978dae07', 'request_id': 'req-0ce24a38-26b9-4232-986f-6b4c9a4faac0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:10.160268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5fe6d64bcd684732a59bc9f0978dae07', 'request_id': 'req-0ce24a38-26b9-4232-986f-6b4c9a4faac0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:10.171947 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5fe6d64bcd684732a59bc9f0978dae07', 'request_id': 'req-0ce24a38-26b9-4232-986f-6b4c9a4faac0'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:10.226018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] The token cache is being invalidate because project 5fe6d64bcd684732a59bc9f0978dae07 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:16:10.227426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:10.228457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:10.233034 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1039/2049] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:16:10 2022] DELETE /identity/v3/projects/5fe6d64bcd684732a59bc9f0978dae07 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:16:10.234848 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0adc91e8-2c3a-4fbd-8d00-412392f639d1 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:10.245748 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0adc91e8-2c3a-4fbd-8d00-412392f639d1 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:10.359480 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce8ac83-a9da-4611-a22b-140c5119d1bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:10.360122 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce8ac83-a9da-4611-a22b-140c5119d1bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:10.360496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce8ac83-a9da-4611-a22b-140c5119d1bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:10.360770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce8ac83-a9da-4611-a22b-140c5119d1bb admin admin] PATH_INFO: `/v3/users/5fbec0379fb84e8980820e7673f2ca4e` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:10.362168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce8ac83-a9da-4611-a22b-140c5119d1bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=5fbec0379fb84e8980820e7673f2ca4e)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:10.366131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce8ac83-a9da-4611-a22b-140c5119d1bb admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:10.445787 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1ce8ac83-a9da-4611-a22b-140c5119d1bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5fbec0379fb84e8980820e7673f2ca4e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:10.446522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1ce8ac83-a9da-4611-a22b-140c5119d1bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5fbec0379fb84e8980820e7673f2ca4e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:10.447098 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1ce8ac83-a9da-4611-a22b-140c5119d1bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5fbec0379fb84e8980820e7673f2ca4e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:10.456925 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1ce8ac83-a9da-4611-a22b-140c5119d1bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5fbec0379fb84e8980820e7673f2ca4e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:10.485709 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1ce8ac83-a9da-4611-a22b-140c5119d1bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5fbec0379fb84e8980820e7673f2ca4e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:10.513666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1011/2050] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:16:10 2022] DELETE /identity/v3/users/5fbec0379fb84e8980820e7673f2ca4e => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:11.338450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:11.342198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:11.389658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce24a38-26b9-4232-986f-6b4c9a4faac0 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:11.537174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:11.537824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:11.538102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:11.538376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] PATH_INFO: `/v3/projects/00296957288c4ad69fc435897be406c2` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:11.539580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] RBAC: Authorizing `identity:delete_project(project_id=00296957288c4ad69fc435897be406c2)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:11.541590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:11.631885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '00296957288c4ad69fc435897be406c2', 'request_id': 'req-6201137b-d133-4f95-ba21-ac68bde1999b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:11.638401 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '00296957288c4ad69fc435897be406c2', 'request_id': 'req-6201137b-d133-4f95-ba21-ac68bde1999b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:11.648344 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '00296957288c4ad69fc435897be406c2', 'request_id': 'req-6201137b-d133-4f95-ba21-ac68bde1999b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:11.663517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] The token cache is being invalidate because project 00296957288c4ad69fc435897be406c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:16:11.666920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:11.670172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:11.673044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1040/2051] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:16:11 2022] DELETE /identity/v3/projects/00296957288c4ad69fc435897be406c2 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:11.677247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce8ac83-a9da-4611-a22b-140c5119d1bb admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:11.679296 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1ce8ac83-a9da-4611-a22b-140c5119d1bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:11.699997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce8ac83-a9da-4611-a22b-140c5119d1bb admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:11.774875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf281ab-2073-4f44-9070-f447fe2aa70e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:11.775868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf281ab-2073-4f44-9070-f447fe2aa70e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:11.776876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf281ab-2073-4f44-9070-f447fe2aa70e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:11.777366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf281ab-2073-4f44-9070-f447fe2aa70e admin admin] PATH_INFO: `/v3/users/58dce294837f4eb9a72e0623d4692ab8` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:11.785473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf281ab-2073-4f44-9070-f447fe2aa70e admin admin] RBAC: Authorizing `identity:delete_user(user_id=58dce294837f4eb9a72e0623d4692ab8)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:11.792182 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf281ab-2073-4f44-9070-f447fe2aa70e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:11.795762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:11.900217 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fdf281ab-2073-4f44-9070-f447fe2aa70e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58dce294837f4eb9a72e0623d4692ab8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:11.901831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fdf281ab-2073-4f44-9070-f447fe2aa70e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58dce294837f4eb9a72e0623d4692ab8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:11.902755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fdf281ab-2073-4f44-9070-f447fe2aa70e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58dce294837f4eb9a72e0623d4692ab8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:11.912852 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fdf281ab-2073-4f44-9070-f447fe2aa70e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58dce294837f4eb9a72e0623d4692ab8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:11.957586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fdf281ab-2073-4f44-9070-f447fe2aa70e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58dce294837f4eb9a72e0623d4692ab8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:11.983300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1012/2052] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:16:11 2022] DELETE /identity/v3/users/58dce294837f4eb9a72e0623d4692ab8 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:12.724697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:12.726611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:12.745044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6201137b-d133-4f95-ba21-ac68bde1999b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:12.757459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:12.833088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ce3700-3c1c-44da-8042-e1987970e50f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:12.834019 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ce3700-3c1c-44da-8042-e1987970e50f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:12.834147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ce3700-3c1c-44da-8042-e1987970e50f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:12.834480 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ce3700-3c1c-44da-8042-e1987970e50f admin admin] PATH_INFO: `/v3/projects/6ba00ace42f14b259c0c308c321669a2` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:12.835742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ce3700-3c1c-44da-8042-e1987970e50f admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ba00ace42f14b259c0c308c321669a2)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:12.837691 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ce3700-3c1c-44da-8042-e1987970e50f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:12.904544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-48ce3700-3c1c-44da-8042-e1987970e50f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ba00ace42f14b259c0c308c321669a2', 'request_id': 'req-48ce3700-3c1c-44da-8042-e1987970e50f'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:12.915609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-48ce3700-3c1c-44da-8042-e1987970e50f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ba00ace42f14b259c0c308c321669a2', 'request_id': 'req-48ce3700-3c1c-44da-8042-e1987970e50f'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:12.929844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-48ce3700-3c1c-44da-8042-e1987970e50f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ba00ace42f14b259c0c308c321669a2', 'request_id': 'req-48ce3700-3c1c-44da-8042-e1987970e50f'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:12.946848 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-48ce3700-3c1c-44da-8042-e1987970e50f admin admin] The token cache is being invalidate because project 6ba00ace42f14b259c0c308c321669a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:16:12.947151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-48ce3700-3c1c-44da-8042-e1987970e50f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:12.947879 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-48ce3700-3c1c-44da-8042-e1987970e50f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:12.949922 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1041/2053] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:16:12 2022] DELETE /identity/v3/projects/6ba00ace42f14b259c0c308c321669a2 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:12.954760 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962c5c8b-2872-440f-ae4c-14adf2032a67 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:12.955283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962c5c8b-2872-440f-ae4c-14adf2032a67 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:12.955370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962c5c8b-2872-440f-ae4c-14adf2032a67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:12.996361 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-962c5c8b-2872-440f-ae4c-14adf2032a67 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:12.998305 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-962c5c8b-2872-440f-ae4c-14adf2032a67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:13.085432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1013/2054] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:13.093864 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd47d4c9-8c0b-4153-87bb-9ab3944939a4 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:13.094259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd47d4c9-8c0b-4153-87bb-9ab3944939a4 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:13.095139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd47d4c9-8c0b-4153-87bb-9ab3944939a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:13.116212 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-bd47d4c9-8c0b-4153-87bb-9ab3944939a4 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:13.117598 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-bd47d4c9-8c0b-4153-87bb-9ab3944939a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:13.126872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1042/2055] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:13.131456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962c5c8b-2872-440f-ae4c-14adf2032a67 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:13.156000 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962c5c8b-2872-440f-ae4c-14adf2032a67 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:13.162561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf4007e-9013-438a-90db-0fdd8e38f47f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:13.163342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf4007e-9013-438a-90db-0fdd8e38f47f admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:13.164010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf4007e-9013-438a-90db-0fdd8e38f47f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:13.164806 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf4007e-9013-438a-90db-0fdd8e38f47f admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:13.167101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf4007e-9013-438a-90db-0fdd8e38f47f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:16:13.167581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf4007e-9013-438a-90db-0fdd8e38f47f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:13.170659 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf4007e-9013-438a-90db-0fdd8e38f47f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:13.193618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd47d4c9-8c0b-4153-87bb-9ab3944939a4 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:13.202128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1014/2056] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:16:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:16:13.206631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd47d4c9-8c0b-4153-87bb-9ab3944939a4 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:13.223300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9ce525-ba94-41de-a7e2-7c287a52427e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:13.224033 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9ce525-ba94-41de-a7e2-7c287a52427e admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:13.224760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9ce525-ba94-41de-a7e2-7c287a52427e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:13.226442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9ce525-ba94-41de-a7e2-7c287a52427e admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:13.227360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9ce525-ba94-41de-a7e2-7c287a52427e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:13.229580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9ce525-ba94-41de-a7e2-7c287a52427e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:13.274301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1043/2057] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:16:13 2022] POST /identity/v3/projects => generated 381 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:16:13.280113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf4007e-9013-438a-90db-0fdd8e38f47f admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:13.295628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf4007e-9013-438a-90db-0fdd8e38f47f admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:13.392768 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f4b8a7-11c1-46d0-af98-9752f5b9671f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:13.393456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f4b8a7-11c1-46d0-af98-9752f5b9671f admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:13.393872 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f4b8a7-11c1-46d0-af98-9752f5b9671f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:13.394207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f4b8a7-11c1-46d0-af98-9752f5b9671f admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:13.394937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f4b8a7-11c1-46d0-af98-9752f5b9671f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:13.396961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f4b8a7-11c1-46d0-af98-9752f5b9671f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:13.411151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:13.430889 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1015/2058] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:16:13 2022] POST /identity/v3/users => generated 291 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:16:13.431993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9ce525-ba94-41de-a7e2-7c287a52427e admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:13.443421 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9ce525-ba94-41de-a7e2-7c287a52427e admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:13.448223 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c835e7b7-db04-4d73-a189-cca30e02dc8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:13.448925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c835e7b7-db04-4d73-a189-cca30e02dc8e admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:13.449281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c835e7b7-db04-4d73-a189-cca30e02dc8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:13.449626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c835e7b7-db04-4d73-a189-cca30e02dc8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:13.450346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c835e7b7-db04-4d73-a189-cca30e02dc8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:13.452068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c835e7b7-db04-4d73-a189-cca30e02dc8e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:13.464351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1044/2059] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:16:13 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:16:13.469202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f4b8a7-11c1-46d0-af98-9752f5b9671f admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:13.488447 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f4b8a7-11c1-46d0-af98-9752f5b9671f admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:13.501546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9e3c7d-125f-4b4d-b75f-80a5f76c76e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:13.502126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9e3c7d-125f-4b4d-b75f-80a5f76c76e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:13.502519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9e3c7d-125f-4b4d-b75f-80a5f76c76e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:13.502894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9e3c7d-125f-4b4d-b75f-80a5f76c76e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:13.503651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9e3c7d-125f-4b4d-b75f-80a5f76c76e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:13.505518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9e3c7d-125f-4b4d-b75f-80a5f76c76e3 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:13.521941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1016/2060] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:16:13 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:16:13.537657 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c835e7b7-db04-4d73-a189-cca30e02dc8e admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:13.553818 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c835e7b7-db04-4d73-a189-cca30e02dc8e admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:13.559264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f2addc-76db-4c5e-9fe6-1d4800fbea5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:13.560045 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f2addc-76db-4c5e-9fe6-1d4800fbea5f admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:13.560484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f2addc-76db-4c5e-9fe6-1d4800fbea5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:13.560864 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f2addc-76db-4c5e-9fe6-1d4800fbea5f admin admin] PATH_INFO: `/v3/projects/9fedaf31a9b64cbfb658837f17204f66/users/7e428194fecb425fb4e1264ce6c49051/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:13.579090 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f2addc-76db-4c5e-9fe6-1d4800fbea5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=9fedaf31a9b64cbfb658837f17204f66, user_id=7e428194fecb425fb4e1264ce6c49051, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:13.582990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f2addc-76db-4c5e-9fe6-1d4800fbea5f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:13.611276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1045/2061] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:16:13 2022] PUT /identity/v3/projects/9fedaf31a9b64cbfb658837f17204f66/users/7e428194fecb425fb4e1264ce6c49051/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:13.616155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbc3145-96b9-4475-93ca-96fe5ca221a0 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:13.616486 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbc3145-96b9-4475-93ca-96fe5ca221a0 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:13.616778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbc3145-96b9-4475-93ca-96fe5ca221a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:13.636845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-9cbc3145-96b9-4475-93ca-96fe5ca221a0 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:13.638674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9cbc3145-96b9-4475-93ca-96fe5ca221a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:13.719772 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1017/2062] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:13.757040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f2addc-76db-4c5e-9fe6-1d4800fbea5f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:13.759447 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f1f2addc-76db-4c5e-9fe6-1d4800fbea5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:13.818603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:13.848568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f2addc-76db-4c5e-9fe6-1d4800fbea5f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:13.853361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecaffff-813e-4cdb-a77c-750c5b775e21 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:13.854067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecaffff-813e-4cdb-a77c-750c5b775e21 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:13.854691 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecaffff-813e-4cdb-a77c-750c5b775e21 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:13.855067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecaffff-813e-4cdb-a77c-750c5b775e21 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:13.867707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecaffff-813e-4cdb-a77c-750c5b775e21 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:13.869268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecaffff-813e-4cdb-a77c-750c5b775e21 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:13.892238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1046/2063] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:16:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:16:14.966603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c0f91f-640f-4e7e-8632-dad43c27f6e3 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:14.967155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c0f91f-640f-4e7e-8632-dad43c27f6e3 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:14.967155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c0f91f-640f-4e7e-8632-dad43c27f6e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:15.029645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-81c0f91f-640f-4e7e-8632-dad43c27f6e3 None None] MFA Rules not processed for user `7e428194fecb425fb4e1264ce6c49051`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:15.032754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-81c0f91f-640f-4e7e-8632-dad43c27f6e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:15.151579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1018/2064] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:14 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:15.163253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecaffff-813e-4cdb-a77c-750c5b775e21 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:15.165341 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-fecaffff-813e-4cdb-a77c-750c5b775e21 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:15.308931 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecaffff-813e-4cdb-a77c-750c5b775e21 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:15.313871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f9e574-9b0a-4828-b2b7-fddc85dece55 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:15.314776 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f9e574-9b0a-4828-b2b7-fddc85dece55 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:15.315239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f9e574-9b0a-4828-b2b7-fddc85dece55 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:15.315683 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f9e574-9b0a-4828-b2b7-fddc85dece55 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:15.327465 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f9e574-9b0a-4828-b2b7-fddc85dece55 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:15.329574 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f9e574-9b0a-4828-b2b7-fddc85dece55 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:15.340442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1047/2065] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:16:15 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:16:16.494063 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb1d1a3-19ce-4586-b07f-f17d1a19161b None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:16.494544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb1d1a3-19ce-4586-b07f-f17d1a19161b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:16.495041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb1d1a3-19ce-4586-b07f-f17d1a19161b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:16.525651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-0eb1d1a3-19ce-4586-b07f-f17d1a19161b None None] MFA Rules not processed for user `4226c61349b440c3a502a7cbc592d883`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:16.527314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0eb1d1a3-19ce-4586-b07f-f17d1a19161b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:16.613529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1019/2066] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:16 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:16.618223 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f9e574-9b0a-4828-b2b7-fddc85dece55 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:16.620859 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-56f9e574-9b0a-4828-b2b7-fddc85dece55 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:16.637362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f9e574-9b0a-4828-b2b7-fddc85dece55 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:16.642353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22742759-48da-437e-a6fc-fecff05e913f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:16.642955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22742759-48da-437e-a6fc-fecff05e913f admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:16.643234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22742759-48da-437e-a6fc-fecff05e913f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:16.643506 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22742759-48da-437e-a6fc-fecff05e913f admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:16.644150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22742759-48da-437e-a6fc-fecff05e913f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:16.646079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22742759-48da-437e-a6fc-fecff05e913f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:16.684711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1048/2067] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:16:16 2022] POST /identity/v3/projects => generated 385 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:16:16.685977 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb1d1a3-19ce-4586-b07f-f17d1a19161b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:16.693846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:16.703947 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb1d1a3-19ce-4586-b07f-f17d1a19161b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:16.781000 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5788565-8a60-4d9d-bbfd-6f96347182a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:16.781605 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5788565-8a60-4d9d-bbfd-6f96347182a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:16.781883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5788565-8a60-4d9d-bbfd-6f96347182a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:16.782149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5788565-8a60-4d9d-bbfd-6f96347182a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:16.782776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5788565-8a60-4d9d-bbfd-6f96347182a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:16.784727 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5788565-8a60-4d9d-bbfd-6f96347182a2 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:16.810923 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1020/2068] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:16:16 2022] POST /identity/v3/users => generated 299 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:16:16.812969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22742759-48da-437e-a6fc-fecff05e913f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:16.831492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22742759-48da-437e-a6fc-fecff05e913f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:16.837683 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66603e8f-72cc-4bf5-819b-78ba53413938 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:16.838268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66603e8f-72cc-4bf5-819b-78ba53413938 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:16.838751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66603e8f-72cc-4bf5-819b-78ba53413938 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:16.839031 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66603e8f-72cc-4bf5-819b-78ba53413938 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:16.839700 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66603e8f-72cc-4bf5-819b-78ba53413938 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:16.842128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66603e8f-72cc-4bf5-819b-78ba53413938 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:16.856217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1049/2069] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:16:16 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:16:16.860609 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5788565-8a60-4d9d-bbfd-6f96347182a2 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:16.877952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5788565-8a60-4d9d-bbfd-6f96347182a2 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:16.891619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e7b55b-ba55-432a-befd-75e207f962e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:16.892528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e7b55b-ba55-432a-befd-75e207f962e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:16.892863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e7b55b-ba55-432a-befd-75e207f962e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:16.893146 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e7b55b-ba55-432a-befd-75e207f962e3 admin admin] PATH_INFO: `/v3/projects/e9bbda2115b847e6ab8ee5f544fc823a/users/ffe6fdd4cbc64f7291975cc8957ba17b/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:16.932124 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e7b55b-ba55-432a-befd-75e207f962e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9bbda2115b847e6ab8ee5f544fc823a, user_id=ffe6fdd4cbc64f7291975cc8957ba17b, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:16.935254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e7b55b-ba55-432a-befd-75e207f962e3 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:16.948784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0d2904-4dc5-41e3-b52e-950146100776 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:16.949972 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0d2904-4dc5-41e3-b52e-950146100776 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:16.950443 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0d2904-4dc5-41e3-b52e-950146100776 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:16.981368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4f0d2904-4dc5-41e3-b52e-950146100776 None None] MFA Rules not processed for user `7e428194fecb425fb4e1264ce6c49051`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:16.982880 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4f0d2904-4dc5-41e3-b52e-950146100776 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:16.984179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1021/2070] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:16:16 2022] PUT /identity/v3/projects/e9bbda2115b847e6ab8ee5f544fc823a/users/ffe6fdd4cbc64f7291975cc8957ba17b/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:16.987679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e7b55b-ba55-432a-befd-75e207f962e3 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:17.061024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1050/2071] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:16 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:16:17.101347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e7b55b-ba55-432a-befd-75e207f962e3 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:17.107330 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1dc84f-8002-4b25-80f4-146582961088 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:17.107943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1dc84f-8002-4b25-80f4-146582961088 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:17.108221 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1dc84f-8002-4b25-80f4-146582961088 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:17.108543 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1dc84f-8002-4b25-80f4-146582961088 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:17.109192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1dc84f-8002-4b25-80f4-146582961088 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:17.110804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1dc84f-8002-4b25-80f4-146582961088 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:17.128374 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1022/2072] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:16:16 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:16:17.137685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0d2904-4dc5-41e3-b52e-950146100776 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:17.157323 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0d2904-4dc5-41e3-b52e-950146100776 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:17.170768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175a56cd-ea55-42f9-9713-c5e2cf42959d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:17.171459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175a56cd-ea55-42f9-9713-c5e2cf42959d admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:17.171748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175a56cd-ea55-42f9-9713-c5e2cf42959d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:17.172024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175a56cd-ea55-42f9-9713-c5e2cf42959d admin admin] PATH_INFO: `/v3/projects/e9bbda2115b847e6ab8ee5f544fc823a/users/ffe6fdd4cbc64f7291975cc8957ba17b/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:17.174039 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175a56cd-ea55-42f9-9713-c5e2cf42959d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9bbda2115b847e6ab8ee5f544fc823a, user_id=ffe6fdd4cbc64f7291975cc8957ba17b, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:17.177732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175a56cd-ea55-42f9-9713-c5e2cf42959d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:17.203025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1051/2073] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:16:17 2022] PUT /identity/v3/projects/e9bbda2115b847e6ab8ee5f544fc823a/users/ffe6fdd4cbc64f7291975cc8957ba17b/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:24.535968 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39237315-9aac-47fd-8079-da2e8f2ed627 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:24.536486 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39237315-9aac-47fd-8079-da2e8f2ed627 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:24.536563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39237315-9aac-47fd-8079-da2e8f2ed627 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:24.567604 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-39237315-9aac-47fd-8079-da2e8f2ed627 None None] MFA Rules not processed for user `ffe6fdd4cbc64f7291975cc8957ba17b`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:24.569241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-39237315-9aac-47fd-8079-da2e8f2ed627 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:24.629796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:24.662894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1023/2074] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:24 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:24.677004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175a56cd-ea55-42f9-9713-c5e2cf42959d admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:24.679442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-175a56cd-ea55-42f9-9713-c5e2cf42959d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:24.735107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:24.753266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175a56cd-ea55-42f9-9713-c5e2cf42959d admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:24.756908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65889e24-35f4-4398-b268-428ec6bef59e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '815504693bcc40b6aef06aacb2a9a35b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:24.757511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65889e24-35f4-4398-b268-428ec6bef59e service glance] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:24.757786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65889e24-35f4-4398-b268-428ec6bef59e service glance] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:24.758054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65889e24-35f4-4398-b268-428ec6bef59e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:24.821815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65889e24-35f4-4398-b268-428ec6bef59e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:24.825261 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65889e24-35f4-4398-b268-428ec6bef59e service glance] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:24.850742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1052/2075] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:16:24 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:16:25.011601 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823fd48a-a098-48ae-8fa8-5f869b31f2ed None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:25.011930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823fd48a-a098-48ae-8fa8-5f869b31f2ed None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:25.012202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823fd48a-a098-48ae-8fa8-5f869b31f2ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:25.036603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-823fd48a-a098-48ae-8fa8-5f869b31f2ed None None] MFA Rules not processed for user `b20a6fb4e7a546058f5fa314c9a59299`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:25.037813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-823fd48a-a098-48ae-8fa8-5f869b31f2ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:25.102052 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1024/2076] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Sep 28 12:16:25 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:25.169434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65889e24-35f4-4398-b268-428ec6bef59e service glance] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:25.171271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-65889e24-35f4-4398-b268-428ec6bef59e service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:25.264744 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65889e24-35f4-4398-b268-428ec6bef59e service glance] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:25.273848 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb33396-522a-472e-9999-d7558d45a435 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e663aad79bb8429aa223564a15332ee2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:25.274500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb33396-522a-472e-9999-d7558d45a435 service swift] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:25.274972 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb33396-522a-472e-9999-d7558d45a435 service swift] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:25.275417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb33396-522a-472e-9999-d7558d45a435 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:25.287318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb33396-522a-472e-9999-d7558d45a435 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:25.290500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb33396-522a-472e-9999-d7558d45a435 service swift] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:25.309968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1053/2077] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Sep 28 12:16:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 28 12:16:31.406052 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f2e7ed-3bd2-47a0-8524-c99f777a1c92 None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:31.406666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f2e7ed-3bd2-47a0-8524-c99f777a1c92 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:31.406666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f2e7ed-3bd2-47a0-8524-c99f777a1c92 None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:31.408532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1025/2078] 10.222.0.26 () {54 vars in 921 bytes} [Wed Sep 28 12:16:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:16:31.473960 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fa2bec-6042-4008-839e-b3f62f718caf None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:31.474567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fa2bec-6042-4008-839e-b3f62f718caf None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:31.474661 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fa2bec-6042-4008-839e-b3f62f718caf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:31.507278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f2e7ed-3bd2-47a0-8524-c99f777a1c92 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:31.509899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e4f2e7ed-3bd2-47a0-8524-c99f777a1c92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:31.540961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-f2fa2bec-6042-4008-839e-b3f62f718caf None None] MFA Rules not processed for user `3758ea0f26264340ad2f154c0111af21`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:31.542298 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f2fa2bec-6042-4008-839e-b3f62f718caf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:31.601272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f2e7ed-3bd2-47a0-8524-c99f777a1c92 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:31.608025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb77cab-ccf3-42d6-a31b-69f7e40bd07b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:31.608973 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb77cab-ccf3-42d6-a31b-69f7e40bd07b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:31.609413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb77cab-ccf3-42d6-a31b-69f7e40bd07b admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:31.609839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb77cab-ccf3-42d6-a31b-69f7e40bd07b admin admin] PATH_INFO: `/v3/users/3b0e4c6131674b3799a374aacdec708f` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:31.611295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb77cab-ccf3-42d6-a31b-69f7e40bd07b admin admin] RBAC: Authorizing `identity:delete_user(user_id=3b0e4c6131674b3799a374aacdec708f)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:31.613736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb77cab-ccf3-42d6-a31b-69f7e40bd07b admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:31.630778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1054/2079] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Sep 28 12:16:31 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:31.643218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fa2bec-6042-4008-839e-b3f62f718caf None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:31.644981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f2fa2bec-6042-4008-839e-b3f62f718caf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:31.663462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fa2bec-6042-4008-839e-b3f62f718caf None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:31.669109 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d3ad74-7b19-4a44-996d-03a7d495c607 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:31.669833 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d3ad74-7b19-4a44-996d-03a7d495c607 service placement] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:31.670221 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d3ad74-7b19-4a44-996d-03a7d495c607 service placement] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:31.670595 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d3ad74-7b19-4a44-996d-03a7d495c607 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:31.678744 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8eb77cab-ccf3-42d6-a31b-69f7e40bd07b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3b0e4c6131674b3799a374aacdec708f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:31.679119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8eb77cab-ccf3-42d6-a31b-69f7e40bd07b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3b0e4c6131674b3799a374aacdec708f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:31.679884 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8eb77cab-ccf3-42d6-a31b-69f7e40bd07b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3b0e4c6131674b3799a374aacdec708f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:31.681081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d3ad74-7b19-4a44-996d-03a7d495c607 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:31.682894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d3ad74-7b19-4a44-996d-03a7d495c607 service placement] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:31.688277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8eb77cab-ccf3-42d6-a31b-69f7e40bd07b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3b0e4c6131674b3799a374aacdec708f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:31.694283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1055/2080] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:16:31 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:16:31.703266 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8eb77cab-ccf3-42d6-a31b-69f7e40bd07b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3b0e4c6131674b3799a374aacdec708f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:31.720541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1026/2081] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:16:31 2022] DELETE /identity/v3/users/3b0e4c6131674b3799a374aacdec708f => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:16:31.927803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d3ad74-7b19-4a44-996d-03a7d495c607 service placement] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:31.945528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d3ad74-7b19-4a44-996d-03a7d495c607 service placement] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:31.951191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:32.059707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc76f79-5479-4b61-a235-c924279177fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:32.060481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc76f79-5479-4b61-a235-c924279177fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:32.060837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc76f79-5479-4b61-a235-c924279177fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:32.061204 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc76f79-5479-4b61-a235-c924279177fe admin admin] PATH_INFO: `/v3/projects/6090ef3dd4494004b0156e49bbeb9273` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:32.062549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc76f79-5479-4b61-a235-c924279177fe admin admin] RBAC: Authorizing `identity:delete_project(project_id=6090ef3dd4494004b0156e49bbeb9273)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:32.064668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc76f79-5479-4b61-a235-c924279177fe admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:32.126889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bbc76f79-5479-4b61-a235-c924279177fe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6090ef3dd4494004b0156e49bbeb9273', 'request_id': 'req-bbc76f79-5479-4b61-a235-c924279177fe'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:32.143450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bbc76f79-5479-4b61-a235-c924279177fe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6090ef3dd4494004b0156e49bbeb9273', 'request_id': 'req-bbc76f79-5479-4b61-a235-c924279177fe'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:32.158371 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bbc76f79-5479-4b61-a235-c924279177fe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6090ef3dd4494004b0156e49bbeb9273', 'request_id': 'req-bbc76f79-5479-4b61-a235-c924279177fe'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:32.182390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bbc76f79-5479-4b61-a235-c924279177fe admin admin] The token cache is being invalidate because project 6090ef3dd4494004b0156e49bbeb9273 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:16:32.182796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bbc76f79-5479-4b61-a235-c924279177fe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:32.186275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bbc76f79-5479-4b61-a235-c924279177fe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:32.192291 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1056/2082] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:16:31 2022] DELETE /identity/v3/projects/6090ef3dd4494004b0156e49bbeb9273 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:32.194002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2c2118-b466-4be6-bcff-4051eeec1514 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:32.194331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2c2118-b466-4be6-bcff-4051eeec1514 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:32.194611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2c2118-b466-4be6-bcff-4051eeec1514 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:32.224859 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-9f2c2118-b466-4be6-bcff-4051eeec1514 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:32.232151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9f2c2118-b466-4be6-bcff-4051eeec1514 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:32.247196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:32.327951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1027/2083] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:32.335395 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257194da-29b6-4d2f-9afe-f0d990a487d8 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:32.336751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257194da-29b6-4d2f-9afe-f0d990a487d8 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:32.336751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257194da-29b6-4d2f-9afe-f0d990a487d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:32.358614 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-257194da-29b6-4d2f-9afe-f0d990a487d8 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:32.359046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-257194da-29b6-4d2f-9afe-f0d990a487d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:32.366984 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1057/2084] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:32.371322 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2c2118-b466-4be6-bcff-4051eeec1514 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:32.381631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2c2118-b466-4be6-bcff-4051eeec1514 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:32.387503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a087ab3f-4d80-41e9-80b6-65edfe4cc18c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:32.388123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a087ab3f-4d80-41e9-80b6-65edfe4cc18c admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:32.388436 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a087ab3f-4d80-41e9-80b6-65edfe4cc18c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:32.388711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a087ab3f-4d80-41e9-80b6-65edfe4cc18c admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:32.389670 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a087ab3f-4d80-41e9-80b6-65edfe4cc18c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:16:32.389986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a087ab3f-4d80-41e9-80b6-65edfe4cc18c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:32.391596 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a087ab3f-4d80-41e9-80b6-65edfe4cc18c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:32.407837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1028/2085] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:16:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:16:32.411841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257194da-29b6-4d2f-9afe-f0d990a487d8 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:32.421831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257194da-29b6-4d2f-9afe-f0d990a487d8 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:32.426876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a770a7b-7a04-4de0-8537-d72c13dec3a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:32.427499 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a770a7b-7a04-4de0-8537-d72c13dec3a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:32.427774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a770a7b-7a04-4de0-8537-d72c13dec3a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:32.428041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a770a7b-7a04-4de0-8537-d72c13dec3a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:32.428740 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a770a7b-7a04-4de0-8537-d72c13dec3a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:32.430630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a770a7b-7a04-4de0-8537-d72c13dec3a5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:32.465257 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1058/2086] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:16:32 2022] POST /identity/v3/projects => generated 385 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:16:32.469164 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a087ab3f-4d80-41e9-80b6-65edfe4cc18c admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:32.483656 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a087ab3f-4d80-41e9-80b6-65edfe4cc18c admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:32.552683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0afd8d6-6773-4fbb-9973-0fb2a15da084 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:32.553292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0afd8d6-6773-4fbb-9973-0fb2a15da084 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:32.553579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0afd8d6-6773-4fbb-9973-0fb2a15da084 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:32.553856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0afd8d6-6773-4fbb-9973-0fb2a15da084 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:32.554490 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0afd8d6-6773-4fbb-9973-0fb2a15da084 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:32.558267 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0afd8d6-6773-4fbb-9973-0fb2a15da084 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:32.581831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1029/2087] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:16:32 2022] POST /identity/v3/users => generated 293 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:16:32.585991 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a770a7b-7a04-4de0-8537-d72c13dec3a5 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:32.597451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a770a7b-7a04-4de0-8537-d72c13dec3a5 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:32.602758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1ff21c-6e28-4bdd-8f02-6780a47562e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:32.603370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1ff21c-6e28-4bdd-8f02-6780a47562e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:32.603652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1ff21c-6e28-4bdd-8f02-6780a47562e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:32.603959 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1ff21c-6e28-4bdd-8f02-6780a47562e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:32.604643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1ff21c-6e28-4bdd-8f02-6780a47562e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:32.606325 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1ff21c-6e28-4bdd-8f02-6780a47562e9 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:32.621232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1059/2088] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:16:32 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:16:32.622266 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0afd8d6-6773-4fbb-9973-0fb2a15da084 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:32.632735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0afd8d6-6773-4fbb-9973-0fb2a15da084 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:32.638109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fd0174-436d-4715-abfb-742b227f2ba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:32.638761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fd0174-436d-4715-abfb-742b227f2ba4 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:32.640685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fd0174-436d-4715-abfb-742b227f2ba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:32.640685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fd0174-436d-4715-abfb-742b227f2ba4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:32.640685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fd0174-436d-4715-abfb-742b227f2ba4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:32.641612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fd0174-436d-4715-abfb-742b227f2ba4 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:32.654120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1030/2089] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:16:32 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:16:32.658622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1ff21c-6e28-4bdd-8f02-6780a47562e9 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:32.672169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1ff21c-6e28-4bdd-8f02-6780a47562e9 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:32.679024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811ed0cb-737f-4a3e-b4a3-75c0ecd24397 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:32.679753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811ed0cb-737f-4a3e-b4a3-75c0ecd24397 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:32.680102 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811ed0cb-737f-4a3e-b4a3-75c0ecd24397 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:32.680488 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811ed0cb-737f-4a3e-b4a3-75c0ecd24397 admin admin] PATH_INFO: `/v3/projects/9b527fca31484a269222cd738b918290/users/f8a5b345d91e40e4aa5e57a8dbaa9c61/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:32.699373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811ed0cb-737f-4a3e-b4a3-75c0ecd24397 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b527fca31484a269222cd738b918290, user_id=f8a5b345d91e40e4aa5e57a8dbaa9c61, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:32.704813 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811ed0cb-737f-4a3e-b4a3-75c0ecd24397 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:32.721553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1060/2090] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:16:32 2022] PUT /identity/v3/projects/9b527fca31484a269222cd738b918290/users/f8a5b345d91e40e4aa5e57a8dbaa9c61/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:32.735319 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fd4397-ca41-4de7-bea7-d9c20b624bdf None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:32.735615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fd4397-ca41-4de7-bea7-d9c20b624bdf None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:32.735875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fd4397-ca41-4de7-bea7-d9c20b624bdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:32.770151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-86fd4397-ca41-4de7-bea7-d9c20b624bdf None None] MFA Rules not processed for user `f8a5b345d91e40e4aa5e57a8dbaa9c61`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:32.771391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-86fd4397-ca41-4de7-bea7-d9c20b624bdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:32.852091 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1031/2091] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:32 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:32.860122 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811ed0cb-737f-4a3e-b4a3-75c0ecd24397 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:32.861932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-811ed0cb-737f-4a3e-b4a3-75c0ecd24397 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:32.906620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:32.956733 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811ed0cb-737f-4a3e-b4a3-75c0ecd24397 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:32.965675 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f69e5e-c060-418c-8a94-7106fc8117a6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:32.966397 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f69e5e-c060-418c-8a94-7106fc8117a6 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:32.966684 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f69e5e-c060-418c-8a94-7106fc8117a6 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:32.966958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f69e5e-c060-418c-8a94-7106fc8117a6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:32.976647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f69e5e-c060-418c-8a94-7106fc8117a6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:32.978072 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f69e5e-c060-418c-8a94-7106fc8117a6 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:32.988388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1061/2092] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:16:32 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:16:37.051898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d98ccc-def6-43da-be11-93557158f3d8 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:37.052520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d98ccc-def6-43da-be11-93557158f3d8 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:37.052520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d98ccc-def6-43da-be11-93557158f3d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:37.071236 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-b7d98ccc-def6-43da-be11-93557158f3d8 None None] MFA Rules not processed for user `4226c61349b440c3a502a7cbc592d883`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:37.072494 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b7d98ccc-def6-43da-be11-93557158f3d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:37.078294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:37.131950 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1032/2093] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:37 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:42.984262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=19 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:47.125021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=20 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:50.593171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc736e5-afa9-4f25-88ea-14e9b3748362 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:50.593696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc736e5-afa9-4f25-88ea-14e9b3748362 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:50.593696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc736e5-afa9-4f25-88ea-14e9b3748362 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:50.615760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-cdc736e5-afa9-4f25-88ea-14e9b3748362 None None] MFA Rules not processed for user `b20a6fb4e7a546058f5fa314c9a59299`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:50.617250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-cdc736e5-afa9-4f25-88ea-14e9b3748362 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:50.681749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1062/2094] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Sep 28 12:16:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:50.694451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d98ccc-def6-43da-be11-93557158f3d8 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:50.696137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b7d98ccc-def6-43da-be11-93557158f3d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:50.778839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d98ccc-def6-43da-be11-93557158f3d8 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:50.784754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0711bb-3cbc-47d9-8020-4e496a562a8c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e663aad79bb8429aa223564a15332ee2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:50.785582 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0711bb-3cbc-47d9-8020-4e496a562a8c service swift] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:50.786015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0711bb-3cbc-47d9-8020-4e496a562a8c service swift] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:50.786419 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0711bb-3cbc-47d9-8020-4e496a562a8c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:50.796197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0711bb-3cbc-47d9-8020-4e496a562a8c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:50.798287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0711bb-3cbc-47d9-8020-4e496a562a8c service swift] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:50.815813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1033/2095] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Sep 28 12:16:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 28 12:16:54.653985 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc736e5-afa9-4f25-88ea-14e9b3748362 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:54.718883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc736e5-afa9-4f25-88ea-14e9b3748362 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:54.724068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8014bcbe-18c7-4971-8430-c25d7e93a093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:54.724933 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8014bcbe-18c7-4971-8430-c25d7e93a093 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:54.725446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8014bcbe-18c7-4971-8430-c25d7e93a093 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:54.725729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8014bcbe-18c7-4971-8430-c25d7e93a093 admin admin] PATH_INFO: `/v3/users/f8a5b345d91e40e4aa5e57a8dbaa9c61` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:54.727061 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8014bcbe-18c7-4971-8430-c25d7e93a093 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f8a5b345d91e40e4aa5e57a8dbaa9c61)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:54.729092 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8014bcbe-18c7-4971-8430-c25d7e93a093 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:54.791071 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8014bcbe-18c7-4971-8430-c25d7e93a093 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f8a5b345d91e40e4aa5e57a8dbaa9c61'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:54.791789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8014bcbe-18c7-4971-8430-c25d7e93a093 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f8a5b345d91e40e4aa5e57a8dbaa9c61'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:54.800108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8014bcbe-18c7-4971-8430-c25d7e93a093 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f8a5b345d91e40e4aa5e57a8dbaa9c61'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:54.800807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8014bcbe-18c7-4971-8430-c25d7e93a093 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f8a5b345d91e40e4aa5e57a8dbaa9c61'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:54.812749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8014bcbe-18c7-4971-8430-c25d7e93a093 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f8a5b345d91e40e4aa5e57a8dbaa9c61'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:54.826633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1063/2096] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:16:54 2022] DELETE /identity/v3/users/f8a5b345d91e40e4aa5e57a8dbaa9c61 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:54.831079 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2316c9c1-476d-40ca-956e-8410765cebad None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:54.831595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2316c9c1-476d-40ca-956e-8410765cebad None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:54.831694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2316c9c1-476d-40ca-956e-8410765cebad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:54.853601 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-2316c9c1-476d-40ca-956e-8410765cebad None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:54.867236 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2316c9c1-476d-40ca-956e-8410765cebad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:54.955534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1034/2097] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:16:54.965269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8014bcbe-18c7-4971-8430-c25d7e93a093 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:54.966692 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8014bcbe-18c7-4971-8430-c25d7e93a093 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:54.996596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:55.050229 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8014bcbe-18c7-4971-8430-c25d7e93a093 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:55.063679 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434e3a9d-a9f8-467f-aab4-45a30bcd967f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:55.064500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434e3a9d-a9f8-467f-aab4-45a30bcd967f service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:55.064767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434e3a9d-a9f8-467f-aab4-45a30bcd967f service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:55.065058 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434e3a9d-a9f8-467f-aab4-45a30bcd967f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:55.077867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434e3a9d-a9f8-467f-aab4-45a30bcd967f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:55.079446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434e3a9d-a9f8-467f-aab4-45a30bcd967f service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:55.090124 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1064/2098] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:16:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:16:55.496076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2316c9c1-476d-40ca-956e-8410765cebad None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:55.497934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2316c9c1-476d-40ca-956e-8410765cebad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:55.518989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2316c9c1-476d-40ca-956e-8410765cebad None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:55.524980 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859b8e09-0cc8-4be2-b351-21d0f030c551 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:55.525693 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859b8e09-0cc8-4be2-b351-21d0f030c551 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:55.526096 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859b8e09-0cc8-4be2-b351-21d0f030c551 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:55.526442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859b8e09-0cc8-4be2-b351-21d0f030c551 admin admin] PATH_INFO: `/v3/projects/9b527fca31484a269222cd738b918290` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:55.528473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859b8e09-0cc8-4be2-b351-21d0f030c551 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9b527fca31484a269222cd738b918290)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:55.530832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859b8e09-0cc8-4be2-b351-21d0f030c551 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:55.584494 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-859b8e09-0cc8-4be2-b351-21d0f030c551 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b527fca31484a269222cd738b918290', 'request_id': 'req-859b8e09-0cc8-4be2-b351-21d0f030c551'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:55.590700 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-859b8e09-0cc8-4be2-b351-21d0f030c551 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b527fca31484a269222cd738b918290', 'request_id': 'req-859b8e09-0cc8-4be2-b351-21d0f030c551'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:55.599705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-859b8e09-0cc8-4be2-b351-21d0f030c551 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b527fca31484a269222cd738b918290', 'request_id': 'req-859b8e09-0cc8-4be2-b351-21d0f030c551'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:55.612460 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-859b8e09-0cc8-4be2-b351-21d0f030c551 admin admin] The token cache is being invalidate because project 9b527fca31484a269222cd738b918290 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:16:55.612749 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-859b8e09-0cc8-4be2-b351-21d0f030c551 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:55.613409 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-859b8e09-0cc8-4be2-b351-21d0f030c551 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:16:55.614896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1035/2099] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:16:55 2022] DELETE /identity/v3/projects/9b527fca31484a269222cd738b918290 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:55.619953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4b599f-5c7b-47c8-b4c3-021438870317 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:55.620269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4b599f-5c7b-47c8-b4c3-021438870317 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:55.620590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4b599f-5c7b-47c8-b4c3-021438870317 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:55.645820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-8f4b599f-5c7b-47c8-b4c3-021438870317 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:55.647926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8f4b599f-5c7b-47c8-b4c3-021438870317 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:55.737404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1065/2100] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:55.740205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efdfbfc-a753-4d64-b39c-0f0b49981b47 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:55.740797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efdfbfc-a753-4d64-b39c-0f0b49981b47 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:55.741152 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efdfbfc-a753-4d64-b39c-0f0b49981b47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:55.762088 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-1efdfbfc-a753-4d64-b39c-0f0b49981b47 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:55.763415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1efdfbfc-a753-4d64-b39c-0f0b49981b47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:55.771740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1036/2101] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:55.776481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4b599f-5c7b-47c8-b4c3-021438870317 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:55.789668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4b599f-5c7b-47c8-b4c3-021438870317 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:55.795092 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a134a270-a489-4f1b-bc40-6e4e2b3d352d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:55.795721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a134a270-a489-4f1b-bc40-6e4e2b3d352d admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:55.795997 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a134a270-a489-4f1b-bc40-6e4e2b3d352d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:55.796284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a134a270-a489-4f1b-bc40-6e4e2b3d352d admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:55.796967 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a134a270-a489-4f1b-bc40-6e4e2b3d352d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:16:55.797263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a134a270-a489-4f1b-bc40-6e4e2b3d352d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:55.798837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a134a270-a489-4f1b-bc40-6e4e2b3d352d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:55.820557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1066/2102] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:16:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:16:55.825656 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efdfbfc-a753-4d64-b39c-0f0b49981b47 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:55.835131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efdfbfc-a753-4d64-b39c-0f0b49981b47 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:55.840503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faeaddfa-8aba-4068-8059-e24c922d7812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:55.841105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faeaddfa-8aba-4068-8059-e24c922d7812 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:55.841384 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faeaddfa-8aba-4068-8059-e24c922d7812 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:55.841655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faeaddfa-8aba-4068-8059-e24c922d7812 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:55.842303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faeaddfa-8aba-4068-8059-e24c922d7812 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:55.844224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faeaddfa-8aba-4068-8059-e24c922d7812 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:55.849199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:55.878889 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1037/2103] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:16:55 2022] POST /identity/v3/projects => generated 381 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:16:55.883118 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a134a270-a489-4f1b-bc40-6e4e2b3d352d admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:55.894147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a134a270-a489-4f1b-bc40-6e4e2b3d352d admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:55.951401 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8868a8-ba19-4bfa-8977-54413917db85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:55.952113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8868a8-ba19-4bfa-8977-54413917db85 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:55.952519 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8868a8-ba19-4bfa-8977-54413917db85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:55.952862 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8868a8-ba19-4bfa-8977-54413917db85 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:55.953653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8868a8-ba19-4bfa-8977-54413917db85 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:55.955640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8868a8-ba19-4bfa-8977-54413917db85 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:55.982608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1067/2104] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:16:55 2022] POST /identity/v3/users => generated 291 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:16:55.984024 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faeaddfa-8aba-4068-8059-e24c922d7812 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:55.994698 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faeaddfa-8aba-4068-8059-e24c922d7812 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:56.000214 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9026d14c-e264-40d0-bc82-48d2b74a68e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:56.000951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9026d14c-e264-40d0-bc82-48d2b74a68e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:56.001323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9026d14c-e264-40d0-bc82-48d2b74a68e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:56.001685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9026d14c-e264-40d0-bc82-48d2b74a68e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:56.002432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9026d14c-e264-40d0-bc82-48d2b74a68e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:56.004140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9026d14c-e264-40d0-bc82-48d2b74a68e1 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:56.016761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1038/2105] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:16:55 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:16:56.020637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8868a8-ba19-4bfa-8977-54413917db85 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:56.031355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8868a8-ba19-4bfa-8977-54413917db85 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:56.036414 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936800c5-18ca-49ce-8620-2137275bb429 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:56.037123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936800c5-18ca-49ce-8620-2137275bb429 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:56.037674 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936800c5-18ca-49ce-8620-2137275bb429 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:56.038046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936800c5-18ca-49ce-8620-2137275bb429 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:56.038797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936800c5-18ca-49ce-8620-2137275bb429 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:56.040524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936800c5-18ca-49ce-8620-2137275bb429 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:56.056549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1068/2106] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:16:56 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:16:56.058542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9026d14c-e264-40d0-bc82-48d2b74a68e1 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:56.072146 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9026d14c-e264-40d0-bc82-48d2b74a68e1 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:56.077448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3e0db9-babc-4780-99b4-c57a13144dcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:56.078136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3e0db9-babc-4780-99b4-c57a13144dcd admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:56.078955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3e0db9-babc-4780-99b4-c57a13144dcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:56.078955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3e0db9-babc-4780-99b4-c57a13144dcd admin admin] PATH_INFO: `/v3/projects/a89923a7c1284bedaf58fcd7097cddae/users/59ab40a82cec482abb11cbbeae448464/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:56.112595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3e0db9-babc-4780-99b4-c57a13144dcd admin admin] RBAC: Authorizing `identity:create_grant(project_id=a89923a7c1284bedaf58fcd7097cddae, user_id=59ab40a82cec482abb11cbbeae448464, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:56.115921 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3e0db9-babc-4780-99b4-c57a13144dcd admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:56.133814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1039/2107] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:16:56 2022] PUT /identity/v3/projects/a89923a7c1284bedaf58fcd7097cddae/users/59ab40a82cec482abb11cbbeae448464/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:56.144790 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c146940d-582f-4594-be43-c4049c528b83 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:56.145304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c146940d-582f-4594-be43-c4049c528b83 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:56.145674 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c146940d-582f-4594-be43-c4049c528b83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:56.192450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-c146940d-582f-4594-be43-c4049c528b83 None None] MFA Rules not processed for user `59ab40a82cec482abb11cbbeae448464`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:56.193777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c146940d-582f-4594-be43-c4049c528b83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:56.219121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:56.279743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1069/2108] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:56 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:16:56.284011 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3e0db9-babc-4780-99b4-c57a13144dcd admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:56.358874 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3e0db9-babc-4780-99b4-c57a13144dcd admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:56.363567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2602be70-0654-48c3-a4db-f253db87771f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:56.364170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2602be70-0654-48c3-a4db-f253db87771f admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:56.364485 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2602be70-0654-48c3-a4db-f253db87771f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:56.364761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2602be70-0654-48c3-a4db-f253db87771f admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:56.365408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2602be70-0654-48c3-a4db-f253db87771f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:56.367313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2602be70-0654-48c3-a4db-f253db87771f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:56.398860 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1040/2109] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:16:56 2022] POST /identity/v3/projects => generated 381 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:16:56.400147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c146940d-582f-4594-be43-c4049c528b83 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:56.411608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c146940d-582f-4594-be43-c4049c528b83 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:56.470725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0d3cec-75dc-4c25-9db4-a28b7ba57d68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:56.471333 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0d3cec-75dc-4c25-9db4-a28b7ba57d68 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:56.471614 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0d3cec-75dc-4c25-9db4-a28b7ba57d68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:56.471885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0d3cec-75dc-4c25-9db4-a28b7ba57d68 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:56.472552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0d3cec-75dc-4c25-9db4-a28b7ba57d68 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:56.474455 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0d3cec-75dc-4c25-9db4-a28b7ba57d68 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:56.536702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1070/2110] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:16:56 2022] POST /identity/v3/users => generated 297 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:16:56.537601 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2602be70-0654-48c3-a4db-f253db87771f admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:56.555699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2602be70-0654-48c3-a4db-f253db87771f admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:56.561864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad3dcf2-1acd-4181-ab75-55f902ea6ce2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:56.563741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad3dcf2-1acd-4181-ab75-55f902ea6ce2 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:56.564150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad3dcf2-1acd-4181-ab75-55f902ea6ce2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:56.564518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad3dcf2-1acd-4181-ab75-55f902ea6ce2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:56.565307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad3dcf2-1acd-4181-ab75-55f902ea6ce2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:56.567059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad3dcf2-1acd-4181-ab75-55f902ea6ce2 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:56.580368 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1041/2111] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:16:56 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:16:56.584174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0d3cec-75dc-4c25-9db4-a28b7ba57d68 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:56.596921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0d3cec-75dc-4c25-9db4-a28b7ba57d68 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:56.601564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09661b77-d1a5-477b-bf4c-acf044de93f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:56.602251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09661b77-d1a5-477b-bf4c-acf044de93f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:56.602560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09661b77-d1a5-477b-bf4c-acf044de93f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:56.602863 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09661b77-d1a5-477b-bf4c-acf044de93f2 admin admin] PATH_INFO: `/v3/projects/e9e2ceef73ba42eaa027b2bb17192e6d/users/bcb5e3d103534756a1a32b7afbfabdd5/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:56.621960 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09661b77-d1a5-477b-bf4c-acf044de93f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9e2ceef73ba42eaa027b2bb17192e6d, user_id=bcb5e3d103534756a1a32b7afbfabdd5, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:56.624912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09661b77-d1a5-477b-bf4c-acf044de93f2 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:56.646137 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1071/2112] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:16:56 2022] PUT /identity/v3/projects/e9e2ceef73ba42eaa027b2bb17192e6d/users/bcb5e3d103534756a1a32b7afbfabdd5/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:16:56.649860 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad3dcf2-1acd-4181-ab75-55f902ea6ce2 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:56.661443 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad3dcf2-1acd-4181-ab75-55f902ea6ce2 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:56.732838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523aa77c-aeb7-40b5-bd38-e8e2e0a9db84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:56.733668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523aa77c-aeb7-40b5-bd38-e8e2e0a9db84 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:56.734050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523aa77c-aeb7-40b5-bd38-e8e2e0a9db84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:56.734601 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523aa77c-aeb7-40b5-bd38-e8e2e0a9db84 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:56.735382 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523aa77c-aeb7-40b5-bd38-e8e2e0a9db84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:56.737638 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523aa77c-aeb7-40b5-bd38-e8e2e0a9db84 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:56.758194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1042/2113] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:16:56 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:16:56.762411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09661b77-d1a5-477b-bf4c-acf044de93f2 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:56.773086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09661b77-d1a5-477b-bf4c-acf044de93f2 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:56.778411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4106602d-e96e-4dd5-8077-8752381d0335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:56.779135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4106602d-e96e-4dd5-8077-8752381d0335 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:56.779350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4106602d-e96e-4dd5-8077-8752381d0335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:56.779628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4106602d-e96e-4dd5-8077-8752381d0335 admin admin] PATH_INFO: `/v3/projects/e9e2ceef73ba42eaa027b2bb17192e6d/users/bcb5e3d103534756a1a32b7afbfabdd5/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:56.781746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4106602d-e96e-4dd5-8077-8752381d0335 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9e2ceef73ba42eaa027b2bb17192e6d, user_id=bcb5e3d103534756a1a32b7afbfabdd5, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:56.784543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4106602d-e96e-4dd5-8077-8752381d0335 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:56.796152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1072/2114] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:16:56 2022] PUT /identity/v3/projects/e9e2ceef73ba42eaa027b2bb17192e6d/users/bcb5e3d103534756a1a32b7afbfabdd5/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:16:56.807782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bbec19-8645-46b8-9bdd-9bc32008e195 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:56.808063 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bbec19-8645-46b8-9bdd-9bc32008e195 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:56.808365 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bbec19-8645-46b8-9bdd-9bc32008e195 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:56.837970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-10bbec19-8645-46b8-9bdd-9bc32008e195 None None] MFA Rules not processed for user `bcb5e3d103534756a1a32b7afbfabdd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:16:56.839721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-10bbec19-8645-46b8-9bdd-9bc32008e195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:56.853347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:16:56.920370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1043/2115] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:16:56 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:16:56.927817 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4106602d-e96e-4dd5-8077-8752381d0335 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:56.929639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4106602d-e96e-4dd5-8077-8752381d0335 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:16:56.994937 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4106602d-e96e-4dd5-8077-8752381d0335 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:56.998699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593e35d8-a166-43b5-b551-e25055e9def5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:56.999316 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e35d8-a166-43b5-b551-e25055e9def5 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:56.999594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e35d8-a166-43b5-b551-e25055e9def5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:56.999867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e35d8-a166-43b5-b551-e25055e9def5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:57.009677 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593e35d8-a166-43b5-b551-e25055e9def5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:57.011145 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593e35d8-a166-43b5-b551-e25055e9def5 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:57.021104 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1073/2116] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:16:56 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:16:57.099753 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bbec19-8645-46b8-9bdd-9bc32008e195 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:16:57.109991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bbec19-8645-46b8-9bdd-9bc32008e195 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:16:57.113592 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca92cc2-cbf2-41fa-a895-a13c4095e826 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:16:57.114186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca92cc2-cbf2-41fa-a895-a13c4095e826 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:16:57.114488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca92cc2-cbf2-41fa-a895-a13c4095e826 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:16:57.114766 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca92cc2-cbf2-41fa-a895-a13c4095e826 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:16:57.171448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca92cc2-cbf2-41fa-a895-a13c4095e826 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:16:57.172872 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca92cc2-cbf2-41fa-a895-a13c4095e826 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:16:57.182290 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1044/2117] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:16:57 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:17:07.017192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=87,keystone:INSERT=5 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:07.178387 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=53 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:20.806463 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593e35d8-a166-43b5-b551-e25055e9def5 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:20.873572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593e35d8-a166-43b5-b551-e25055e9def5 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:20.878107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd7f50b-9ce3-4c33-91af-d8f073509adb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:20.878731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd7f50b-9ce3-4c33-91af-d8f073509adb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:20.879006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd7f50b-9ce3-4c33-91af-d8f073509adb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:20.879274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd7f50b-9ce3-4c33-91af-d8f073509adb admin admin] PATH_INFO: `/v3/users/59ab40a82cec482abb11cbbeae448464` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:20.880430 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd7f50b-9ce3-4c33-91af-d8f073509adb admin admin] RBAC: Authorizing `identity:delete_user(user_id=59ab40a82cec482abb11cbbeae448464)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:20.882329 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd7f50b-9ce3-4c33-91af-d8f073509adb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:20.938886 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-dcd7f50b-9ce3-4c33-91af-d8f073509adb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '59ab40a82cec482abb11cbbeae448464'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:20.939754 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-dcd7f50b-9ce3-4c33-91af-d8f073509adb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '59ab40a82cec482abb11cbbeae448464'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:20.949866 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-dcd7f50b-9ce3-4c33-91af-d8f073509adb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '59ab40a82cec482abb11cbbeae448464'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:20.950694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-dcd7f50b-9ce3-4c33-91af-d8f073509adb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '59ab40a82cec482abb11cbbeae448464'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:20.963710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-dcd7f50b-9ce3-4c33-91af-d8f073509adb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '59ab40a82cec482abb11cbbeae448464'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:20.980389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1074/2118] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:17:20 2022] DELETE /identity/v3/users/59ab40a82cec482abb11cbbeae448464 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:17:20.983588 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368219a4-e94e-4b2c-940f-fe5d0d604aaa None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:20.984065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368219a4-e94e-4b2c-940f-fe5d0d604aaa None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:20.984289 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368219a4-e94e-4b2c-940f-fe5d0d604aaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:21.006332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-368219a4-e94e-4b2c-940f-fe5d0d604aaa None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:21.007545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-368219a4-e94e-4b2c-940f-fe5d0d604aaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:21.074283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1045/2119] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:17:21.084111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd7f50b-9ce3-4c33-91af-d8f073509adb admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:21.085544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-dcd7f50b-9ce3-4c33-91af-d8f073509adb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:21.144972 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd7f50b-9ce3-4c33-91af-d8f073509adb admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:21.148650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213eb86f-20c4-43b9-9961-df2925b45a51 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:21.149251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213eb86f-20c4-43b9-9961-df2925b45a51 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:21.149528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213eb86f-20c4-43b9-9961-df2925b45a51 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:21.149799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213eb86f-20c4-43b9-9961-df2925b45a51 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:21.158873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213eb86f-20c4-43b9-9961-df2925b45a51 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:21.160362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213eb86f-20c4-43b9-9961-df2925b45a51 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:21.169568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1075/2120] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:17:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:17:21.444595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368219a4-e94e-4b2c-940f-fe5d0d604aaa None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:21.446331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-368219a4-e94e-4b2c-940f-fe5d0d604aaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:21.458186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368219a4-e94e-4b2c-940f-fe5d0d604aaa None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:21.462863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:21.463985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:21.464259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:21.464563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] PATH_INFO: `/v3/projects/a89923a7c1284bedaf58fcd7097cddae` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:21.465776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a89923a7c1284bedaf58fcd7097cddae)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:21.467704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:21.517684 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a89923a7c1284bedaf58fcd7097cddae', 'request_id': 'req-e20dff27-43a3-42bf-8691-52a47d6ca946'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:21.523110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a89923a7c1284bedaf58fcd7097cddae', 'request_id': 'req-e20dff27-43a3-42bf-8691-52a47d6ca946'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:21.530395 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a89923a7c1284bedaf58fcd7097cddae', 'request_id': 'req-e20dff27-43a3-42bf-8691-52a47d6ca946'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:21.541355 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] The token cache is being invalidate because project a89923a7c1284bedaf58fcd7097cddae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:17:21.542158 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:21.542792 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:21.544269 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1046/2121] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:17:21 2022] DELETE /identity/v3/projects/a89923a7c1284bedaf58fcd7097cddae => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:17:21.548430 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213eb86f-20c4-43b9-9961-df2925b45a51 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:21.558527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213eb86f-20c4-43b9-9961-df2925b45a51 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:21.576069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:21.615516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f157508f-5abf-40b5-9b79-996b35e03d83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:21.616129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f157508f-5abf-40b5-9b79-996b35e03d83 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:21.616464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f157508f-5abf-40b5-9b79-996b35e03d83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:21.616764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f157508f-5abf-40b5-9b79-996b35e03d83 admin admin] PATH_INFO: `/v3/users/bcb5e3d103534756a1a32b7afbfabdd5` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:21.617872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f157508f-5abf-40b5-9b79-996b35e03d83 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bcb5e3d103534756a1a32b7afbfabdd5)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:21.619959 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f157508f-5abf-40b5-9b79-996b35e03d83 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:21.663715 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f157508f-5abf-40b5-9b79-996b35e03d83 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bcb5e3d103534756a1a32b7afbfabdd5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:21.664387 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f157508f-5abf-40b5-9b79-996b35e03d83 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bcb5e3d103534756a1a32b7afbfabdd5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:21.672057 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f157508f-5abf-40b5-9b79-996b35e03d83 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bcb5e3d103534756a1a32b7afbfabdd5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:21.672850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f157508f-5abf-40b5-9b79-996b35e03d83 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bcb5e3d103534756a1a32b7afbfabdd5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:21.689149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f157508f-5abf-40b5-9b79-996b35e03d83 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bcb5e3d103534756a1a32b7afbfabdd5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:21.702817 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1076/2122] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:17:21 2022] DELETE /identity/v3/users/bcb5e3d103534756a1a32b7afbfabdd5 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:17:22.051188 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:22.052768 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:22.073888 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20dff27-43a3-42bf-8691-52a47d6ca946 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:22.136691 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c4b2d5-f3c2-48f7-a7de-42039029be29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:22.137395 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c4b2d5-f3c2-48f7-a7de-42039029be29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:22.137716 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c4b2d5-f3c2-48f7-a7de-42039029be29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:22.137992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c4b2d5-f3c2-48f7-a7de-42039029be29 admin admin] PATH_INFO: `/v3/projects/e9e2ceef73ba42eaa027b2bb17192e6d` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:22.139327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c4b2d5-f3c2-48f7-a7de-42039029be29 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9e2ceef73ba42eaa027b2bb17192e6d)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:22.141587 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c4b2d5-f3c2-48f7-a7de-42039029be29 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:22.176731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:22.199881 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-d6c4b2d5-f3c2-48f7-a7de-42039029be29 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9e2ceef73ba42eaa027b2bb17192e6d', 'request_id': 'req-d6c4b2d5-f3c2-48f7-a7de-42039029be29'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:22.207727 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-d6c4b2d5-f3c2-48f7-a7de-42039029be29 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9e2ceef73ba42eaa027b2bb17192e6d', 'request_id': 'req-d6c4b2d5-f3c2-48f7-a7de-42039029be29'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:22.216886 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-d6c4b2d5-f3c2-48f7-a7de-42039029be29 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9e2ceef73ba42eaa027b2bb17192e6d', 'request_id': 'req-d6c4b2d5-f3c2-48f7-a7de-42039029be29'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:22.229225 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-d6c4b2d5-f3c2-48f7-a7de-42039029be29 admin admin] The token cache is being invalidate because project e9e2ceef73ba42eaa027b2bb17192e6d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:17:22.229523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-d6c4b2d5-f3c2-48f7-a7de-42039029be29 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:22.230196 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-d6c4b2d5-f3c2-48f7-a7de-42039029be29 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:22.232231 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1047/2123] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:17:22 2022] DELETE /identity/v3/projects/e9e2ceef73ba42eaa027b2bb17192e6d => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:17:22.236751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917d0d1f-9100-49f5-83f5-d8f65f0ea0cc None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:22.237309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917d0d1f-9100-49f5-83f5-d8f65f0ea0cc None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:22.237309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917d0d1f-9100-49f5-83f5-d8f65f0ea0cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:22.258174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-917d0d1f-9100-49f5-83f5-d8f65f0ea0cc None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:22.259405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-917d0d1f-9100-49f5-83f5-d8f65f0ea0cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:22.318130 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1077/2124] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:17:22.326526 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2a223d-1250-4b47-94d5-06ea08587285 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:22.326814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2a223d-1250-4b47-94d5-06ea08587285 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:22.327089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2a223d-1250-4b47-94d5-06ea08587285 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:22.346865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-5e2a223d-1250-4b47-94d5-06ea08587285 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:22.348073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5e2a223d-1250-4b47-94d5-06ea08587285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:22.356299 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1048/2125] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:17:22.360396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917d0d1f-9100-49f5-83f5-d8f65f0ea0cc None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:22.369924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917d0d1f-9100-49f5-83f5-d8f65f0ea0cc None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:22.374463 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0dc5a9-ac94-4296-af1e-5c6f34e2218f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:22.375054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0dc5a9-ac94-4296-af1e-5c6f34e2218f admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:22.375347 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0dc5a9-ac94-4296-af1e-5c6f34e2218f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:22.375604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0dc5a9-ac94-4296-af1e-5c6f34e2218f admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:22.376278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0dc5a9-ac94-4296-af1e-5c6f34e2218f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:17:22.376615 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0dc5a9-ac94-4296-af1e-5c6f34e2218f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:22.378181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0dc5a9-ac94-4296-af1e-5c6f34e2218f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:22.389866 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1078/2126] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:17:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:17:22.393878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2a223d-1250-4b47-94d5-06ea08587285 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:22.403403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2a223d-1250-4b47-94d5-06ea08587285 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:22.408383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3019c33e-37b0-4e3b-9edf-e9e22ff3ebf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:22.408979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3019c33e-37b0-4e3b-9edf-e9e22ff3ebf8 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:22.409261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3019c33e-37b0-4e3b-9edf-e9e22ff3ebf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:22.409532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3019c33e-37b0-4e3b-9edf-e9e22ff3ebf8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:22.410189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3019c33e-37b0-4e3b-9edf-e9e22ff3ebf8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:22.412134 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3019c33e-37b0-4e3b-9edf-e9e22ff3ebf8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:22.440941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1049/2127] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:17:22 2022] POST /identity/v3/projects => generated 387 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:17:22.445217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0dc5a9-ac94-4296-af1e-5c6f34e2218f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:22.455472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0dc5a9-ac94-4296-af1e-5c6f34e2218f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:22.483877 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:22.517961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7ecc25-7153-4009-9740-75d3177e2b3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:22.518568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7ecc25-7153-4009-9740-75d3177e2b3b admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:22.518969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7ecc25-7153-4009-9740-75d3177e2b3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:22.519172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7ecc25-7153-4009-9740-75d3177e2b3b admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:22.519842 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7ecc25-7153-4009-9740-75d3177e2b3b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:22.522028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7ecc25-7153-4009-9740-75d3177e2b3b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:22.544364 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1079/2128] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:17:22 2022] POST /identity/v3/users => generated 294 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:17:22.548522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3019c33e-37b0-4e3b-9edf-e9e22ff3ebf8 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:22.559232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3019c33e-37b0-4e3b-9edf-e9e22ff3ebf8 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:22.563832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadc8dda-597e-4cc5-8be3-e8c771b49486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:22.564449 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadc8dda-597e-4cc5-8be3-e8c771b49486 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:22.564730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadc8dda-597e-4cc5-8be3-e8c771b49486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:22.565001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadc8dda-597e-4cc5-8be3-e8c771b49486 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:22.565640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eadc8dda-597e-4cc5-8be3-e8c771b49486 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:22.567261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eadc8dda-597e-4cc5-8be3-e8c771b49486 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:22.580007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1050/2129] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:17:22 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:17:22.584637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7ecc25-7153-4009-9740-75d3177e2b3b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:22.597120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7ecc25-7153-4009-9740-75d3177e2b3b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:22.602244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a03e9d-d3db-403b-9e2d-bd2197daaf57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:22.602828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a03e9d-d3db-403b-9e2d-bd2197daaf57 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:22.603103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a03e9d-d3db-403b-9e2d-bd2197daaf57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:22.603373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a03e9d-d3db-403b-9e2d-bd2197daaf57 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:22.604076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a03e9d-d3db-403b-9e2d-bd2197daaf57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:22.605696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a03e9d-d3db-403b-9e2d-bd2197daaf57 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:22.616323 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1080/2130] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:17:22 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:17:22.621055 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadc8dda-597e-4cc5-8be3-e8c771b49486 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:22.630901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadc8dda-597e-4cc5-8be3-e8c771b49486 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:22.636280 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d51a428-82bd-4efc-88b1-13bae6c958eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:22.636961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d51a428-82bd-4efc-88b1-13bae6c958eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:22.637239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d51a428-82bd-4efc-88b1-13bae6c958eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:22.637511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d51a428-82bd-4efc-88b1-13bae6c958eb admin admin] PATH_INFO: `/v3/projects/af25dd4e115e45d58d10c1a019007979/users/f58caf5c93c443aaa67252beff6a35b9/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:22.651614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d51a428-82bd-4efc-88b1-13bae6c958eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=af25dd4e115e45d58d10c1a019007979, user_id=f58caf5c93c443aaa67252beff6a35b9, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:22.654275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d51a428-82bd-4efc-88b1-13bae6c958eb admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:22.667891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1051/2131] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:17:22 2022] PUT /identity/v3/projects/af25dd4e115e45d58d10c1a019007979/users/f58caf5c93c443aaa67252beff6a35b9/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:17:22.679327 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8495f3ac-851f-40a1-94c1-818fbc6b46f1 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:22.679613 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8495f3ac-851f-40a1-94c1-818fbc6b46f1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:22.679885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8495f3ac-851f-40a1-94c1-818fbc6b46f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:22.708915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-8495f3ac-851f-40a1-94c1-818fbc6b46f1 None None] MFA Rules not processed for user `f58caf5c93c443aaa67252beff6a35b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:22.710113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8495f3ac-851f-40a1-94c1-818fbc6b46f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:22.773106 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1081/2132] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:22 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:17:22.777588 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d51a428-82bd-4efc-88b1-13bae6c958eb admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:22.838583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d51a428-82bd-4efc-88b1-13bae6c958eb admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:22.843443 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a6771a-a2a7-4e80-9a40-5faebde5aa81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:22.844041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a6771a-a2a7-4e80-9a40-5faebde5aa81 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:22.844335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a6771a-a2a7-4e80-9a40-5faebde5aa81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:22.844613 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a6771a-a2a7-4e80-9a40-5faebde5aa81 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:22.845512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a6771a-a2a7-4e80-9a40-5faebde5aa81 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:22.847405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a6771a-a2a7-4e80-9a40-5faebde5aa81 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:22.889580 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1052/2133] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:17:22 2022] POST /identity/v3/projects => generated 389 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:17:22.893367 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8495f3ac-851f-40a1-94c1-818fbc6b46f1 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:22.904862 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8495f3ac-851f-40a1-94c1-818fbc6b46f1 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:22.970428 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93237225-d384-4600-9247-da21fb29a87a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:22.971150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93237225-d384-4600-9247-da21fb29a87a admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:22.971488 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93237225-d384-4600-9247-da21fb29a87a admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:22.971848 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93237225-d384-4600-9247-da21fb29a87a admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:22.972584 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93237225-d384-4600-9247-da21fb29a87a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:22.974575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93237225-d384-4600-9247-da21fb29a87a admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:23.012965 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1082/2134] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:17:22 2022] POST /identity/v3/users => generated 301 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:17:23.014297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a6771a-a2a7-4e80-9a40-5faebde5aa81 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:23.028937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a6771a-a2a7-4e80-9a40-5faebde5aa81 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:23.039183 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94916bf6-c847-4c8d-929b-46f2fb2ac425 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:23.039877 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94916bf6-c847-4c8d-929b-46f2fb2ac425 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:23.040225 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94916bf6-c847-4c8d-929b-46f2fb2ac425 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:23.040588 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94916bf6-c847-4c8d-929b-46f2fb2ac425 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:23.041338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94916bf6-c847-4c8d-929b-46f2fb2ac425 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:23.043533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94916bf6-c847-4c8d-929b-46f2fb2ac425 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:23.060045 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93237225-d384-4600-9247-da21fb29a87a admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:23.065046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1053/2135] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:17:23 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:17:23.075226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93237225-d384-4600-9247-da21fb29a87a admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:23.081601 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe82171-5d91-4c84-ad36-4a5b31b68852 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:23.082260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe82171-5d91-4c84-ad36-4a5b31b68852 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:23.082583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe82171-5d91-4c84-ad36-4a5b31b68852 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:23.082805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe82171-5d91-4c84-ad36-4a5b31b68852 admin admin] PATH_INFO: `/v3/projects/399ca919995f4920aceb6730017114c2/users/4a2bfed541f94222a3d51a5cddbffe2a/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:23.098624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe82171-5d91-4c84-ad36-4a5b31b68852 admin admin] RBAC: Authorizing `identity:create_grant(project_id=399ca919995f4920aceb6730017114c2, user_id=4a2bfed541f94222a3d51a5cddbffe2a, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:23.101303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe82171-5d91-4c84-ad36-4a5b31b68852 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:23.115045 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1083/2136] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:17:23 2022] PUT /identity/v3/projects/399ca919995f4920aceb6730017114c2/users/4a2bfed541f94222a3d51a5cddbffe2a/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:17:23.118991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94916bf6-c847-4c8d-929b-46f2fb2ac425 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:23.129348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94916bf6-c847-4c8d-929b-46f2fb2ac425 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:23.148567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:23.186746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039ec567-c5d6-4360-b928-a8a8a701bd99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:23.187335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039ec567-c5d6-4360-b928-a8a8a701bd99 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:23.187614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039ec567-c5d6-4360-b928-a8a8a701bd99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:23.187884 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039ec567-c5d6-4360-b928-a8a8a701bd99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:23.188564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039ec567-c5d6-4360-b928-a8a8a701bd99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:23.190156 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039ec567-c5d6-4360-b928-a8a8a701bd99 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:23.200147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1054/2137] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:17:23 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:17:23.205973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe82171-5d91-4c84-ad36-4a5b31b68852 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:23.211764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:23.217904 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe82171-5d91-4c84-ad36-4a5b31b68852 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:23.225018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef114d6-74fc-4e2c-af60-dc231f4ec71c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:23.226103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef114d6-74fc-4e2c-af60-dc231f4ec71c admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:23.226326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef114d6-74fc-4e2c-af60-dc231f4ec71c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:23.226845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef114d6-74fc-4e2c-af60-dc231f4ec71c admin admin] PATH_INFO: `/v3/projects/399ca919995f4920aceb6730017114c2/users/4a2bfed541f94222a3d51a5cddbffe2a/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:23.230719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef114d6-74fc-4e2c-af60-dc231f4ec71c admin admin] RBAC: Authorizing `identity:create_grant(project_id=399ca919995f4920aceb6730017114c2, user_id=4a2bfed541f94222a3d51a5cddbffe2a, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:23.233943 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef114d6-74fc-4e2c-af60-dc231f4ec71c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:23.245438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1084/2138] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:17:23 2022] PUT /identity/v3/projects/399ca919995f4920aceb6730017114c2/users/4a2bfed541f94222a3d51a5cddbffe2a/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:17:23.255571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995e8570-ae43-42cf-83ed-8c3af8aa3f4a None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:23.255856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995e8570-ae43-42cf-83ed-8c3af8aa3f4a None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:23.256129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995e8570-ae43-42cf-83ed-8c3af8aa3f4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:23.284151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-995e8570-ae43-42cf-83ed-8c3af8aa3f4a None None] MFA Rules not processed for user `4a2bfed541f94222a3d51a5cddbffe2a`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:23.285383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-995e8570-ae43-42cf-83ed-8c3af8aa3f4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:23.378095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1055/2139] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:23 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:17:23.390955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef114d6-74fc-4e2c-af60-dc231f4ec71c admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:23.392901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0ef114d6-74fc-4e2c-af60-dc231f4ec71c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:23.453454 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef114d6-74fc-4e2c-af60-dc231f4ec71c admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:23.457028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef25723-c8b3-47b0-961b-351bc31359fb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:23.457618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef25723-c8b3-47b0-961b-351bc31359fb service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:23.457891 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef25723-c8b3-47b0-961b-351bc31359fb service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:23.458158 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef25723-c8b3-47b0-961b-351bc31359fb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:23.466781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef25723-c8b3-47b0-961b-351bc31359fb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:23.468181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef25723-c8b3-47b0-961b-351bc31359fb service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:23.477513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1085/2140] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:17:23 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:17:23.610721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995e8570-ae43-42cf-83ed-8c3af8aa3f4a None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:23.672211 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995e8570-ae43-42cf-83ed-8c3af8aa3f4a None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:23.678032 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbc97a9-8235-4e45-8e7c-2adb2c0a72b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:23.678691 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbc97a9-8235-4e45-8e7c-2adb2c0a72b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:23.678995 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbc97a9-8235-4e45-8e7c-2adb2c0a72b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:23.679287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbc97a9-8235-4e45-8e7c-2adb2c0a72b1 admin admin] PATH_INFO: `/v3/users/f58caf5c93c443aaa67252beff6a35b9` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:23.680439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbc97a9-8235-4e45-8e7c-2adb2c0a72b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f58caf5c93c443aaa67252beff6a35b9)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:23.683001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbc97a9-8235-4e45-8e7c-2adb2c0a72b1 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:23.732066 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7dbc97a9-8235-4e45-8e7c-2adb2c0a72b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f58caf5c93c443aaa67252beff6a35b9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:23.732747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7dbc97a9-8235-4e45-8e7c-2adb2c0a72b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f58caf5c93c443aaa67252beff6a35b9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:23.733414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7dbc97a9-8235-4e45-8e7c-2adb2c0a72b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f58caf5c93c443aaa67252beff6a35b9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:23.734840 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:23.742692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7dbc97a9-8235-4e45-8e7c-2adb2c0a72b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f58caf5c93c443aaa67252beff6a35b9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:23.755012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7dbc97a9-8235-4e45-8e7c-2adb2c0a72b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f58caf5c93c443aaa67252beff6a35b9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:23.767896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1056/2141] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:17:23 2022] DELETE /identity/v3/users/f58caf5c93c443aaa67252beff6a35b9 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:17:23.772605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60610e75-c8c2-4e0d-b6a7-7bc0ddf6a25c None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:23.772938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60610e75-c8c2-4e0d-b6a7-7bc0ddf6a25c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:23.773222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60610e75-c8c2-4e0d-b6a7-7bc0ddf6a25c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:23.793198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-60610e75-c8c2-4e0d-b6a7-7bc0ddf6a25c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:23.794420 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-60610e75-c8c2-4e0d-b6a7-7bc0ddf6a25c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:23.858756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1086/2142] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:17:23.868420 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbc97a9-8235-4e45-8e7c-2adb2c0a72b1 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:23.871000 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-7dbc97a9-8235-4e45-8e7c-2adb2c0a72b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:23.931598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbc97a9-8235-4e45-8e7c-2adb2c0a72b1 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:23.936213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521094d3-3dcb-4e61-bbd2-2e8a761d8f45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:23.936851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521094d3-3dcb-4e61-bbd2-2e8a761d8f45 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:23.937128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521094d3-3dcb-4e61-bbd2-2e8a761d8f45 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:23.937400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521094d3-3dcb-4e61-bbd2-2e8a761d8f45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:23.947143 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521094d3-3dcb-4e61-bbd2-2e8a761d8f45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:23.948799 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521094d3-3dcb-4e61-bbd2-2e8a761d8f45 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:23.958225 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1057/2143] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:17:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:17:24.330187 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60610e75-c8c2-4e0d-b6a7-7bc0ddf6a25c None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:24.332514 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-60610e75-c8c2-4e0d-b6a7-7bc0ddf6a25c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:24.345559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60610e75-c8c2-4e0d-b6a7-7bc0ddf6a25c None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:24.350158 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:24.350785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:24.351080 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:24.351381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] PATH_INFO: `/v3/projects/af25dd4e115e45d58d10c1a019007979` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:24.352483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=af25dd4e115e45d58d10c1a019007979)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:24.354945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:24.417373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af25dd4e115e45d58d10c1a019007979', 'request_id': 'req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.426309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af25dd4e115e45d58d10c1a019007979', 'request_id': 'req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.437541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af25dd4e115e45d58d10c1a019007979', 'request_id': 'req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.448970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521094d3-3dcb-4e61-bbd2-2e8a761d8f45 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:24.450923 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-521094d3-3dcb-4e61-bbd2-2e8a761d8f45 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:24.452958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] The token cache is being invalidate because project af25dd4e115e45d58d10c1a019007979 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:17:24.453331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.454481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.459862 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1087/2144] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:17:24 2022] DELETE /identity/v3/projects/af25dd4e115e45d58d10c1a019007979 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:17:24.461152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:24.462600 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:24.471188 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521094d3-3dcb-4e61-bbd2-2e8a761d8f45 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:24.492957 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:24.559268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921b6953-6c04-4c21-b7ec-a9c15b80a8a5 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:24.564270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0d01f6-9093-4f83-b67e-77fb301f9888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:24.565003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0d01f6-9093-4f83-b67e-77fb301f9888 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:24.565356 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0d01f6-9093-4f83-b67e-77fb301f9888 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:24.565700 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0d01f6-9093-4f83-b67e-77fb301f9888 admin admin] PATH_INFO: `/v3/users/4a2bfed541f94222a3d51a5cddbffe2a` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:24.567449 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0d01f6-9093-4f83-b67e-77fb301f9888 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a2bfed541f94222a3d51a5cddbffe2a)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:24.570003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0d01f6-9093-4f83-b67e-77fb301f9888 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:24.582174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739abd4b-d590-4676-b772-a34ccc22dffd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:24.583603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739abd4b-d590-4676-b772-a34ccc22dffd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:24.583603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739abd4b-d590-4676-b772-a34ccc22dffd admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:24.583797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739abd4b-d590-4676-b772-a34ccc22dffd admin admin] PATH_INFO: `/v3/users/ca2c40f0872e4601b28e365f882a0adf` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:24.585418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739abd4b-d590-4676-b772-a34ccc22dffd admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca2c40f0872e4601b28e365f882a0adf)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:24.587860 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739abd4b-d590-4676-b772-a34ccc22dffd admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:24.646670 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1b0d01f6-9093-4f83-b67e-77fb301f9888 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a2bfed541f94222a3d51a5cddbffe2a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.648742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1b0d01f6-9093-4f83-b67e-77fb301f9888 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a2bfed541f94222a3d51a5cddbffe2a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.657108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1b0d01f6-9093-4f83-b67e-77fb301f9888 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a2bfed541f94222a3d51a5cddbffe2a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.657732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1b0d01f6-9093-4f83-b67e-77fb301f9888 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a2bfed541f94222a3d51a5cddbffe2a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.658176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-739abd4b-d590-4676-b772-a34ccc22dffd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca2c40f0872e4601b28e365f882a0adf'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.658983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-739abd4b-d590-4676-b772-a34ccc22dffd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca2c40f0872e4601b28e365f882a0adf'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.659947 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-739abd4b-d590-4676-b772-a34ccc22dffd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca2c40f0872e4601b28e365f882a0adf'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.667444 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-739abd4b-d590-4676-b772-a34ccc22dffd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca2c40f0872e4601b28e365f882a0adf'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.670349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1b0d01f6-9093-4f83-b67e-77fb301f9888 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a2bfed541f94222a3d51a5cddbffe2a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.679479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:24.681782 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-739abd4b-d590-4676-b772-a34ccc22dffd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca2c40f0872e4601b28e365f882a0adf'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:24.685666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1088/2145] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:17:24 2022] DELETE /identity/v3/users/4a2bfed541f94222a3d51a5cddbffe2a => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:17:24.696241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1058/2146] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:17:24 2022] DELETE /identity/v3/users/ca2c40f0872e4601b28e365f882a0adf => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:17:24.701313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faac49f0-bc41-4b3b-b3ef-95690fdedd7b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:24.701606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faac49f0-bc41-4b3b-b3ef-95690fdedd7b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:24.701881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faac49f0-bc41-4b3b-b3ef-95690fdedd7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:24.722321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-faac49f0-bc41-4b3b-b3ef-95690fdedd7b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:24.723770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-faac49f0-bc41-4b3b-b3ef-95690fdedd7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:24.787800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1089/2147] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:17:24.797604 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739abd4b-d590-4676-b772-a34ccc22dffd admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:24.799045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-739abd4b-d590-4676-b772-a34ccc22dffd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:24.866558 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739abd4b-d590-4676-b772-a34ccc22dffd admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:24.871333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8f310e-4ab9-4f82-a99e-226a9d45b1c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:24.872007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8f310e-4ab9-4f82-a99e-226a9d45b1c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:24.872332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8f310e-4ab9-4f82-a99e-226a9d45b1c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:24.872736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8f310e-4ab9-4f82-a99e-226a9d45b1c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:24.882582 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8f310e-4ab9-4f82-a99e-226a9d45b1c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:24.884069 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8f310e-4ab9-4f82-a99e-226a9d45b1c7 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:24.894872 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1059/2148] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:17:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:17:25.012149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faac49f0-bc41-4b3b-b3ef-95690fdedd7b None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:25.013730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-faac49f0-bc41-4b3b-b3ef-95690fdedd7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:25.025619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faac49f0-bc41-4b3b-b3ef-95690fdedd7b None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:25.030754 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:25.031401 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.031666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.031961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] PATH_INFO: `/v3/projects/399ca919995f4920aceb6730017114c2` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.033097 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] RBAC: Authorizing `identity:delete_project(project_id=399ca919995f4920aceb6730017114c2)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:25.035202 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:25.076581 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:25.103315 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '399ca919995f4920aceb6730017114c2', 'request_id': 'req-45f07df6-a2c9-4057-a852-38d9079d8831'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:25.110734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '399ca919995f4920aceb6730017114c2', 'request_id': 'req-45f07df6-a2c9-4057-a852-38d9079d8831'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:25.121027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '399ca919995f4920aceb6730017114c2', 'request_id': 'req-45f07df6-a2c9-4057-a852-38d9079d8831'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:25.134574 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] The token cache is being invalidate because project 399ca919995f4920aceb6730017114c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:17:25.135095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:25.135801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:25.137566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1090/2149] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:17:25 2022] DELETE /identity/v3/projects/399ca919995f4920aceb6730017114c2 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:17:25.142463 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38aeb735-86b3-4406-a5c9-85c73b9d44da None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.142747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38aeb735-86b3-4406-a5c9-85c73b9d44da None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.143018 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38aeb735-86b3-4406-a5c9-85c73b9d44da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.163800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-38aeb735-86b3-4406-a5c9-85c73b9d44da None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:25.165047 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-38aeb735-86b3-4406-a5c9-85c73b9d44da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:25.240925 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1060/2150] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:17:25.241253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:25.241253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:25.242428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2468cac5-c946-42b7-9d25-cb8ed62ad9f2 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.242722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2468cac5-c946-42b7-9d25-cb8ed62ad9f2 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.244244 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2468cac5-c946-42b7-9d25-cb8ed62ad9f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.253283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f07df6-a2c9-4057-a852-38d9079d8831 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:25.266496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f53352a-52a2-4929-84a4-f1b809dccbbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:25.267380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f53352a-52a2-4929-84a4-f1b809dccbbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.267667 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f53352a-52a2-4929-84a4-f1b809dccbbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.268228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f53352a-52a2-4929-84a4-f1b809dccbbd admin admin] PATH_INFO: `/v3/projects/5f63c1760e3642d4b6a3f5c499832072` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.269847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f53352a-52a2-4929-84a4-f1b809dccbbd admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f63c1760e3642d4b6a3f5c499832072)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:25.271879 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f53352a-52a2-4929-84a4-f1b809dccbbd admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:25.274998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-2468cac5-c946-42b7-9d25-cb8ed62ad9f2 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:25.276259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2468cac5-c946-42b7-9d25-cb8ed62ad9f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:25.285849 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1061/2151] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:17:25.290199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2468cac5-c946-42b7-9d25-cb8ed62ad9f2 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:25.300059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2468cac5-c946-42b7-9d25-cb8ed62ad9f2 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:25.305054 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666a3df8-9904-4518-9267-f2ee64833cef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:25.305660 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666a3df8-9904-4518-9267-f2ee64833cef admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.305940 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666a3df8-9904-4518-9267-f2ee64833cef admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.306212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666a3df8-9904-4518-9267-f2ee64833cef admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.306931 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666a3df8-9904-4518-9267-f2ee64833cef admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:17:25.307791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666a3df8-9904-4518-9267-f2ee64833cef admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:25.309442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666a3df8-9904-4518-9267-f2ee64833cef admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:25.323547 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1062/2152] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:17:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:17:25.328230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666a3df8-9904-4518-9267-f2ee64833cef admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:25.350232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8f53352a-52a2-4929-84a4-f1b809dccbbd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f63c1760e3642d4b6a3f5c499832072', 'request_id': 'req-8f53352a-52a2-4929-84a4-f1b809dccbbd'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:25.356649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666a3df8-9904-4518-9267-f2ee64833cef admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:25.357756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8f53352a-52a2-4929-84a4-f1b809dccbbd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f63c1760e3642d4b6a3f5c499832072', 'request_id': 'req-8f53352a-52a2-4929-84a4-f1b809dccbbd'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:25.370719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8f53352a-52a2-4929-84a4-f1b809dccbbd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f63c1760e3642d4b6a3f5c499832072', 'request_id': 'req-8f53352a-52a2-4929-84a4-f1b809dccbbd'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:25.390661 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8f53352a-52a2-4929-84a4-f1b809dccbbd admin admin] The token cache is being invalidate because project 5f63c1760e3642d4b6a3f5c499832072 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:17:25.390960 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8f53352a-52a2-4929-84a4-f1b809dccbbd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:25.392494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8f53352a-52a2-4929-84a4-f1b809dccbbd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:17:25.397729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1091/2153] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:17:25 2022] DELETE /identity/v3/projects/5f63c1760e3642d4b6a3f5c499832072 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:17:25.399559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed8f78c-a13c-40dc-9821-b485265675cf None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.399876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed8f78c-a13c-40dc-9821-b485265675cf None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.400233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed8f78c-a13c-40dc-9821-b485265675cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.422405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-9ed8f78c-a13c-40dc-9821-b485265675cf None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:25.423735 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-9ed8f78c-a13c-40dc-9821-b485265675cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:25.455407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d16ac8-f01a-4764-bb36-78eb3d9d4086 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:25.456672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d16ac8-f01a-4764-bb36-78eb3d9d4086 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.457453 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d16ac8-f01a-4764-bb36-78eb3d9d4086 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.457724 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d16ac8-f01a-4764-bb36-78eb3d9d4086 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.458804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d16ac8-f01a-4764-bb36-78eb3d9d4086 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:25.460750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d16ac8-f01a-4764-bb36-78eb3d9d4086 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:25.500440 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:DELETE=2,keystone:SELECT=96,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:25.521094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1063/2154] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:17:25 2022] POST /identity/v3/projects => generated 373 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:17:25.522810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1092/2155] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:17:25.523022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d16ac8-f01a-4764-bb36-78eb3d9d4086 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:25.527202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-26d16ac8-f01a-4764-bb36-78eb3d9d4086 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:25.530459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f63c925-fb57-4aa7-a26c-3a52ed1b1cb0 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.530846 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f63c925-fb57-4aa7-a26c-3a52ed1b1cb0 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.531230 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f63c925-fb57-4aa7-a26c-3a52ed1b1cb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.554489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-9f63c925-fb57-4aa7-a26c-3a52ed1b1cb0 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:25.555864 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-9f63c925-fb57-4aa7-a26c-3a52ed1b1cb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:25.610693 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:25.623256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d16ac8-f01a-4764-bb36-78eb3d9d4086 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:25.627907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b695d5-0d52-40af-ad33-6d272ece214e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:25.628531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b695d5-0d52-40af-ad33-6d272ece214e admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.628805 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b695d5-0d52-40af-ad33-6d272ece214e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.629072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b695d5-0d52-40af-ad33-6d272ece214e admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.629696 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b695d5-0d52-40af-ad33-6d272ece214e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:25.631605 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b695d5-0d52-40af-ad33-6d272ece214e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:25.638292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1093/2156] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:17:25.639281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f63c925-fb57-4aa7-a26c-3a52ed1b1cb0 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:25.651435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f63c925-fb57-4aa7-a26c-3a52ed1b1cb0 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:25.658301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1064/2157] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:17:25 2022] POST /identity/v3/users => generated 287 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:17:25.659303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b695d5-0d52-40af-ad33-6d272ece214e admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:25.659876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffd27d3-7c33-4b90-8061-65337115bf84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:25.662049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffd27d3-7c33-4b90-8061-65337115bf84 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.662338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffd27d3-7c33-4b90-8061-65337115bf84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.662617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffd27d3-7c33-4b90-8061-65337115bf84 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.663343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffd27d3-7c33-4b90-8061-65337115bf84 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:17:25.663653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffd27d3-7c33-4b90-8061-65337115bf84 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:25.665496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffd27d3-7c33-4b90-8061-65337115bf84 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:25.669584 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b695d5-0d52-40af-ad33-6d272ece214e admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:25.674583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad9b71a-2ba7-4a7d-b8de-4c2f8d8ee9d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:25.675246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad9b71a-2ba7-4a7d-b8de-4c2f8d8ee9d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.675590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad9b71a-2ba7-4a7d-b8de-4c2f8d8ee9d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.675935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad9b71a-2ba7-4a7d-b8de-4c2f8d8ee9d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.676700 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad9b71a-2ba7-4a7d-b8de-4c2f8d8ee9d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:25.678356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad9b71a-2ba7-4a7d-b8de-4c2f8d8ee9d8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:25.684362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1094/2158] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:17:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:17:25.685321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffd27d3-7c33-4b90-8061-65337115bf84 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:25.696541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1065/2159] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:17:25 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:17:25.696747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffd27d3-7c33-4b90-8061-65337115bf84 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:25.697755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad9b71a-2ba7-4a7d-b8de-4c2f8d8ee9d8 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:25.702679 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51741238-2514-4f54-8493-7037091a2cc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:25.703686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51741238-2514-4f54-8493-7037091a2cc4 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.704115 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51741238-2514-4f54-8493-7037091a2cc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.704638 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51741238-2514-4f54-8493-7037091a2cc4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.705220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51741238-2514-4f54-8493-7037091a2cc4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:25.707121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51741238-2514-4f54-8493-7037091a2cc4 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:25.708255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad9b71a-2ba7-4a7d-b8de-4c2f8d8ee9d8 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:25.713754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f2cfc8-f1f5-4576-986f-4c6e01a6388c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:25.714433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f2cfc8-f1f5-4576-986f-4c6e01a6388c admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.714779 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f2cfc8-f1f5-4576-986f-4c6e01a6388c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.715117 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f2cfc8-f1f5-4576-986f-4c6e01a6388c admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.715820 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f2cfc8-f1f5-4576-986f-4c6e01a6388c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:25.720548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f2cfc8-f1f5-4576-986f-4c6e01a6388c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:25.739949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1066/2160] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:17:25 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:17:25.740994 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f2cfc8-f1f5-4576-986f-4c6e01a6388c admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:25.748210 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1095/2161] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:17:25 2022] POST /identity/v3/projects => generated 373 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:17:25.752675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f2cfc8-f1f5-4576-986f-4c6e01a6388c admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:25.753225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51741238-2514-4f54-8493-7037091a2cc4 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:25.825247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51741238-2514-4f54-8493-7037091a2cc4 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:25.827187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b97a09a-5532-43f8-a69d-1ccf5c4abdbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:25.827960 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b97a09a-5532-43f8-a69d-1ccf5c4abdbf admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.828460 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b97a09a-5532-43f8-a69d-1ccf5c4abdbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.828832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b97a09a-5532-43f8-a69d-1ccf5c4abdbf admin admin] PATH_INFO: `/v3/projects/1d5209c6077a4fd592a983d1430f630b/users/2b3682aef6f8415f9a5cac2db2f7bff9/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.836630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46fb681-bef8-41b8-8972-2f19c47454fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:25.837296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46fb681-bef8-41b8-8972-2f19c47454fe admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.837579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46fb681-bef8-41b8-8972-2f19c47454fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.837849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46fb681-bef8-41b8-8972-2f19c47454fe admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.839005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46fb681-bef8-41b8-8972-2f19c47454fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:25.840938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46fb681-bef8-41b8-8972-2f19c47454fe admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:25.845560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b97a09a-5532-43f8-a69d-1ccf5c4abdbf admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d5209c6077a4fd592a983d1430f630b, user_id=2b3682aef6f8415f9a5cac2db2f7bff9, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:25.848238 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b97a09a-5532-43f8-a69d-1ccf5c4abdbf admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:25.862927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1067/2162] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:17:25 2022] PUT /identity/v3/projects/1d5209c6077a4fd592a983d1430f630b/users/2b3682aef6f8415f9a5cac2db2f7bff9/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:17:25.863410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1096/2163] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:17:25 2022] POST /identity/v3/users => generated 287 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:17:25.867354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b97a09a-5532-43f8-a69d-1ccf5c4abdbf admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:25.867876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f207a14-97d9-4500-a9b9-b2b5fbbf1d5b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.868160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f207a14-97d9-4500-a9b9-b2b5fbbf1d5b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.868502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f207a14-97d9-4500-a9b9-b2b5fbbf1d5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.886833 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-8f207a14-97d9-4500-a9b9-b2b5fbbf1d5b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:25.888018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8f207a14-97d9-4500-a9b9-b2b5fbbf1d5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:25.917406 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:25.932818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b97a09a-5532-43f8-a69d-1ccf5c4abdbf admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:25.937889 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4966a1d4-a159-4734-b76d-2eb56d64ef69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:25.938761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4966a1d4-a159-4734-b76d-2eb56d64ef69 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.938761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4966a1d4-a159-4734-b76d-2eb56d64ef69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.939048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4966a1d4-a159-4734-b76d-2eb56d64ef69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.939747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4966a1d4-a159-4734-b76d-2eb56d64ef69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:25.941401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4966a1d4-a159-4734-b76d-2eb56d64ef69 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:25.951245 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1068/2164] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:17:25 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:17:25.953434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1097/2165] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:17:25.955277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4966a1d4-a159-4734-b76d-2eb56d64ef69 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:25.962707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f207a14-97d9-4500-a9b9-b2b5fbbf1d5b None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:25.964626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8f207a14-97d9-4500-a9b9-b2b5fbbf1d5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:25.966907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4966a1d4-a159-4734-b76d-2eb56d64ef69 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:25.972791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2726b0d4-a431-4ec7-b582-d16f205e808e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:25.973394 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2726b0d4-a431-4ec7-b582-d16f205e808e admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:25.973667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2726b0d4-a431-4ec7-b582-d16f205e808e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:25.973935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2726b0d4-a431-4ec7-b582-d16f205e808e admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:25.975101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2726b0d4-a431-4ec7-b582-d16f205e808e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:25.977674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2726b0d4-a431-4ec7-b582-d16f205e808e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:25.988994 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1069/2166] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:17:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:17:25.993120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2726b0d4-a431-4ec7-b582-d16f205e808e admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:26.002584 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2726b0d4-a431-4ec7-b582-d16f205e808e admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:26.007180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c79d0c-8e75-40f1-aa46-2e5f8e8e4c52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:26.007816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c79d0c-8e75-40f1-aa46-2e5f8e8e4c52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:26.008100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c79d0c-8e75-40f1-aa46-2e5f8e8e4c52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:26.008395 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c79d0c-8e75-40f1-aa46-2e5f8e8e4c52 admin admin] PATH_INFO: `/v3/projects/b3720873acbd42708f9ab86b749dae3d/users/6d9e364f2e12418d8d53d8c0dd272b38/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:26.011294 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:26.023473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c79d0c-8e75-40f1-aa46-2e5f8e8e4c52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3720873acbd42708f9ab86b749dae3d, user_id=6d9e364f2e12418d8d53d8c0dd272b38, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:26.026869 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c79d0c-8e75-40f1-aa46-2e5f8e8e4c52 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:26.029468 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f207a14-97d9-4500-a9b9-b2b5fbbf1d5b None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:26.033897 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b787a11c-e71f-4a42-b951-176213fcf3a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:26.034591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b787a11c-e71f-4a42-b951-176213fcf3a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:26.035099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b787a11c-e71f-4a42-b951-176213fcf3a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:26.035381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b787a11c-e71f-4a42-b951-176213fcf3a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:26.040050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1070/2167] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:17:25 2022] PUT /identity/v3/projects/b3720873acbd42708f9ab86b749dae3d/users/6d9e364f2e12418d8d53d8c0dd272b38/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:17:26.045754 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b787a11c-e71f-4a42-b951-176213fcf3a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:26.047222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b787a11c-e71f-4a42-b951-176213fcf3a5 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:26.051889 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad4570-4a51-4ae6-affc-6101344252f7 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:26.052180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad4570-4a51-4ae6-affc-6101344252f7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:26.052574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad4570-4a51-4ae6-affc-6101344252f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:26.057166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1098/2168] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:17:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:17:26.081139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-88ad4570-4a51-4ae6-affc-6101344252f7 None None] MFA Rules not processed for user `6d9e364f2e12418d8d53d8c0dd272b38`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:26.082450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-88ad4570-4a51-4ae6-affc-6101344252f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:26.149640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1071/2169] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:26 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:17:26.161559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b787a11c-e71f-4a42-b951-176213fcf3a5 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:26.162965 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b787a11c-e71f-4a42-b951-176213fcf3a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:26.222460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b787a11c-e71f-4a42-b951-176213fcf3a5 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:26.278338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a134d61-3004-4b52-a810-65c0a4fe70be service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:26.278942 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a134d61-3004-4b52-a810-65c0a4fe70be service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:26.279215 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a134d61-3004-4b52-a810-65c0a4fe70be service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:26.279497 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a134d61-3004-4b52-a810-65c0a4fe70be service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:26.287745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a134d61-3004-4b52-a810-65c0a4fe70be service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:26.289285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a134d61-3004-4b52-a810-65c0a4fe70be service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:26.298705 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1099/2170] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:17:26 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:17:26.793159 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2887f46-bf40-41be-92de-337a0fcd178b None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:26.793648 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2887f46-bf40-41be-92de-337a0fcd178b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:26.793743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2887f46-bf40-41be-92de-337a0fcd178b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:26.824048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-a2887f46-bf40-41be-92de-337a0fcd178b None None] MFA Rules not processed for user `2b3682aef6f8415f9a5cac2db2f7bff9`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:26.825363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a2887f46-bf40-41be-92de-337a0fcd178b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:26.902466 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1072/2171] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:26 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:17:26.913485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a134d61-3004-4b52-a810-65c0a4fe70be service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:17:26.915427 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2a134d61-3004-4b52-a810-65c0a4fe70be service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:26.926558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a134d61-3004-4b52-a810-65c0a4fe70be service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:17:26.931285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6778dcb3-34df-42d1-bd96-79f2debc82f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:17:26.932184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6778dcb3-34df-42d1-bd96-79f2debc82f7 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:26.932660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6778dcb3-34df-42d1-bd96-79f2debc82f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:26.933094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6778dcb3-34df-42d1-bd96-79f2debc82f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:26.942224 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6778dcb3-34df-42d1-bd96-79f2debc82f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:17:26.943654 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6778dcb3-34df-42d1-bd96-79f2debc82f7 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:17:26.954121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1100/2172] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:17:26 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:17:27.708163 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cc2110-0cef-48e3-9fe8-046d53d6c655 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:27.708471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cc2110-0cef-48e3-9fe8-046d53d6c655 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:27.708751 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cc2110-0cef-48e3-9fe8-046d53d6c655 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:27.727629 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-f2cc2110-0cef-48e3-9fe8-046d53d6c655 None None] MFA Rules not processed for user `6d9e364f2e12418d8d53d8c0dd272b38`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:27.729400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f2cc2110-0cef-48e3-9fe8-046d53d6c655 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:27.737641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1073/2173] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:27 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:17:28.335622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38398e5a-8aa6-4b80-a5f5-f9bbef6b0bef None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:17:28.336158 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38398e5a-8aa6-4b80-a5f5-f9bbef6b0bef None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:17:28.336158 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38398e5a-8aa6-4b80-a5f5-f9bbef6b0bef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:17:28.357562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-38398e5a-8aa6-4b80-a5f5-f9bbef6b0bef None None] MFA Rules not processed for user `2b3682aef6f8415f9a5cac2db2f7bff9`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:17:28.359469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-38398e5a-8aa6-4b80-a5f5-f9bbef6b0bef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:17:28.368353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1101/2174] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:17:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:17:37.722180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=90,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:17:38.350726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=64 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:18:22.449394 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cc2110-0cef-48e3-9fe8-046d53d6c655 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:18:22.506571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cc2110-0cef-48e3-9fe8-046d53d6c655 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:18:22.510364 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d27472-2c77-4a95-98ee-0087107ac721 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:18:22.510972 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d27472-2c77-4a95-98ee-0087107ac721 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:18:22.511251 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d27472-2c77-4a95-98ee-0087107ac721 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:18:22.511520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d27472-2c77-4a95-98ee-0087107ac721 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:18:22.513283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-51d27472-2c77-4a95-98ee-0087107ac721 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:18:22.581602 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d27472-2c77-4a95-98ee-0087107ac721 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:18:22.583294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d27472-2c77-4a95-98ee-0087107ac721 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:18:22.593683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1074/2175] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:18:22 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:18:22.639174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51786629-15e3-4cad-b4b1-3320462e8534 None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:18:22.639673 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51786629-15e3-4cad-b4b1-3320462e8534 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:18:22.639870 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51786629-15e3-4cad-b4b1-3320462e8534 None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:18:22.641446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1102/2176] 10.222.0.26 () {54 vars in 961 bytes} [Wed Sep 28 12:18:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:18:22.647405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86751a03-82d1-4fde-8c04-0274f8419301 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:18:22.647831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86751a03-82d1-4fde-8c04-0274f8419301 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:18:22.648108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86751a03-82d1-4fde-8c04-0274f8419301 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:18:22.689675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-86751a03-82d1-4fde-8c04-0274f8419301 None None] MFA Rules not processed for user `3758ea0f26264340ad2f154c0111af21`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:18:22.690933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-86751a03-82d1-4fde-8c04-0274f8419301 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:18:22.784231 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1075/2177] 10.222.0.26 () {58 vars in 1058 bytes} [Wed Sep 28 12:18:22 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:18:22.785730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51786629-15e3-4cad-b4b1-3320462e8534 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:18:22.811187 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51786629-15e3-4cad-b4b1-3320462e8534 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:18:22.816434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f31565-f4ba-43f7-85c0-364164d31b98 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:18:22.817039 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f31565-f4ba-43f7-85c0-364164d31b98 service placement] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:18:22.817319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f31565-f4ba-43f7-85c0-364164d31b98 service placement] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:18:22.817589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f31565-f4ba-43f7-85c0-364164d31b98 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:18:22.819360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c6f31565-f4ba-43f7-85c0-364164d31b98 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:18:22.831954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f31565-f4ba-43f7-85c0-364164d31b98 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:18:22.833589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f31565-f4ba-43f7-85c0-364164d31b98 service placement] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:18:22.847452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1103/2178] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:18:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:18:32.764294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=79 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:18:32.840790 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=6 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:18:44.764063 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f349dfbe-9216-4664-ad4a-47a4a23e2a41 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:18:44.764618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f349dfbe-9216-4664-ad4a-47a4a23e2a41 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:18:44.764723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f349dfbe-9216-4664-ad4a-47a4a23e2a41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:18:44.784132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-f349dfbe-9216-4664-ad4a-47a4a23e2a41 None None] MFA Rules not processed for user `b20a6fb4e7a546058f5fa314c9a59299`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:18:44.785359 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f349dfbe-9216-4664-ad4a-47a4a23e2a41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:18:44.838325 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1076/2179] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Sep 28 12:18:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:18:44.878269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f31565-f4ba-43f7-85c0-364164d31b98 service placement] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:18:44.879694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c6f31565-f4ba-43f7-85c0-364164d31b98 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:18:44.945352 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f31565-f4ba-43f7-85c0-364164d31b98 service placement] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:18:44.951145 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2915d8-3d45-4da4-abcc-c85aab659164 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e663aad79bb8429aa223564a15332ee2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:18:44.951774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2915d8-3d45-4da4-abcc-c85aab659164 service swift] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:18:44.952051 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2915d8-3d45-4da4-abcc-c85aab659164 service swift] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:18:44.952891 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2915d8-3d45-4da4-abcc-c85aab659164 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:18:44.961244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2915d8-3d45-4da4-abcc-c85aab659164 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:18:44.962915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2915d8-3d45-4da4-abcc-c85aab659164 service swift] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:18:44.972167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1104/2180] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Sep 28 12:18:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 28 12:18:45.464606 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769c4796-8b4a-43bc-bd90-d448ec802d86 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:18:45.464893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769c4796-8b4a-43bc-bd90-d448ec802d86 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:18:45.465173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769c4796-8b4a-43bc-bd90-d448ec802d86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:18:45.484799 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-769c4796-8b4a-43bc-bd90-d448ec802d86 None None] MFA Rules not processed for user `b20a6fb4e7a546058f5fa314c9a59299`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:18:45.485997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-769c4796-8b4a-43bc-bd90-d448ec802d86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:18:45.493108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1077/2181] 10.222.0.26 () {58 vars in 967 bytes} [Wed Sep 28 12:18:45 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:18:45.506591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2915d8-3d45-4da4-abcc-c85aab659164 service swift] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:18:45.516883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2915d8-3d45-4da4-abcc-c85aab659164 service swift] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:18:45.521985 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e71c91-5d55-4530-b2aa-19c1c95feb99 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e663aad79bb8429aa223564a15332ee2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:18:45.522572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e71c91-5d55-4530-b2aa-19c1c95feb99 service swift] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:18:45.522850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e71c91-5d55-4530-b2aa-19c1c95feb99 service swift] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:18:45.523120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e71c91-5d55-4530-b2aa-19c1c95feb99 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:18:45.531048 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e71c91-5d55-4530-b2aa-19c1c95feb99 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:18:45.532654 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e71c91-5d55-4530-b2aa-19c1c95feb99 service swift] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:18:45.540666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1105/2182] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Sep 28 12:18:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 28 12:18:45.778010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c8dbac-72bb-4a72-b3fc-a9d1719d469d None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:18:45.778476 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c8dbac-72bb-4a72-b3fc-a9d1719d469d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:18:45.778655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c8dbac-72bb-4a72-b3fc-a9d1719d469d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:18:45.799174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-19c8dbac-72bb-4a72-b3fc-a9d1719d469d None None] MFA Rules not processed for user `b20a6fb4e7a546058f5fa314c9a59299`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:18:45.800648 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-19c8dbac-72bb-4a72-b3fc-a9d1719d469d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:18:45.807697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1078/2183] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Sep 28 12:18:45 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:18:45.820336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e71c91-5d55-4530-b2aa-19c1c95feb99 service swift] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:18:45.831153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e71c91-5d55-4530-b2aa-19c1c95feb99 service swift] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:18:45.836147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8077d5ea-b816-494d-9e8c-133ae44ea95a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e663aad79bb8429aa223564a15332ee2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:18:45.836762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8077d5ea-b816-494d-9e8c-133ae44ea95a service swift] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:18:45.837044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8077d5ea-b816-494d-9e8c-133ae44ea95a service swift] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:18:45.837316 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8077d5ea-b816-494d-9e8c-133ae44ea95a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:18:45.845624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8077d5ea-b816-494d-9e8c-133ae44ea95a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:18:45.847457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8077d5ea-b816-494d-9e8c-133ae44ea95a service swift] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:18:45.855954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1106/2184] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Sep 28 12:18:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 28 12:18:46.185454 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fd484e-bec0-4d22-ab60-4b4a9f2cc28e None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:18:46.185806 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fd484e-bec0-4d22-ab60-4b4a9f2cc28e None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:18:46.186147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fd484e-bec0-4d22-ab60-4b4a9f2cc28e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:18:46.207504 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-d6fd484e-bec0-4d22-ab60-4b4a9f2cc28e None None] MFA Rules not processed for user `b20a6fb4e7a546058f5fa314c9a59299`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:18:46.208742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d6fd484e-bec0-4d22-ab60-4b4a9f2cc28e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:18:46.216670 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1079/2185] 10.222.0.26 () {58 vars in 967 bytes} [Wed Sep 28 12:18:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:18:46.229255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8077d5ea-b816-494d-9e8c-133ae44ea95a service swift] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:18:46.239952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8077d5ea-b816-494d-9e8c-133ae44ea95a service swift] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:18:46.244987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1513f0a-b8a7-4e7f-805e-2bd346a5bc3d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e663aad79bb8429aa223564a15332ee2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:18:46.245580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1513f0a-b8a7-4e7f-805e-2bd346a5bc3d service swift] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:18:46.245861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1513f0a-b8a7-4e7f-805e-2bd346a5bc3d service swift] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:18:46.246135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1513f0a-b8a7-4e7f-805e-2bd346a5bc3d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:18:46.253991 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1513f0a-b8a7-4e7f-805e-2bd346a5bc3d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:18:46.255562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1513f0a-b8a7-4e7f-805e-2bd346a5bc3d service swift] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:18:46.263946 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1107/2186] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Sep 28 12:18:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 28 12:18:56.201461 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=34 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:18:56.260219 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=50 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:19:38.392133 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3242cd-398f-4245-938e-56c31af17745 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:38.392768 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3242cd-398f-4245-938e-56c31af17745 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:38.392768 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3242cd-398f-4245-938e-56c31af17745 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:38.412012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-cf3242cd-398f-4245-938e-56c31af17745 None None] MFA Rules not processed for user `2b3682aef6f8415f9a5cac2db2f7bff9`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:19:38.413284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-cf3242cd-398f-4245-938e-56c31af17745 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:19:38.422386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1080/2187] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:19:38 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:19:48.406068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:19:49.388147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1513f0a-b8a7-4e7f-805e-2bd346a5bc3d service swift] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:19:49.397340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1513f0a-b8a7-4e7f-805e-2bd346a5bc3d service swift] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:19:49.401095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6df5bc-4df8-4152-9e70-73f5358ae4f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:19:49.401712 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6df5bc-4df8-4152-9e70-73f5358ae4f4 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:49.401988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6df5bc-4df8-4152-9e70-73f5358ae4f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:49.402256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6df5bc-4df8-4152-9e70-73f5358ae4f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:49.404505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-cb6df5bc-4df8-4152-9e70-73f5358ae4f4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:19:49.461820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6df5bc-4df8-4152-9e70-73f5358ae4f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:19:49.463172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6df5bc-4df8-4152-9e70-73f5358ae4f4 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:19:49.472816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1108/2188] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:19:49 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:19:50.987216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3242cd-398f-4245-938e-56c31af17745 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:19:50.997072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3242cd-398f-4245-938e-56c31af17745 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:19:51.000639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e86979-6dfc-42d5-8103-9d3ab18bc1eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:19:51.001234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e86979-6dfc-42d5-8103-9d3ab18bc1eb service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:51.001512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e86979-6dfc-42d5-8103-9d3ab18bc1eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:51.001775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e86979-6dfc-42d5-8103-9d3ab18bc1eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:51.003503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-79e86979-6dfc-42d5-8103-9d3ab18bc1eb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:19:51.081013 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e86979-6dfc-42d5-8103-9d3ab18bc1eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:19:51.082521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e86979-6dfc-42d5-8103-9d3ab18bc1eb service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:19:51.094168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1081/2189] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:19:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:19:55.636569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6df5bc-4df8-4152-9e70-73f5358ae4f4 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:19:55.645171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6df5bc-4df8-4152-9e70-73f5358ae4f4 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:19:55.648714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e920bd6-241d-44ae-b456-59f84a2d7438 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:19:55.649302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e920bd6-241d-44ae-b456-59f84a2d7438 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:55.649576 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e920bd6-241d-44ae-b456-59f84a2d7438 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:55.649843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e920bd6-241d-44ae-b456-59f84a2d7438 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:55.651537 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7e920bd6-241d-44ae-b456-59f84a2d7438 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:19:55.710005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e920bd6-241d-44ae-b456-59f84a2d7438 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:19:55.711473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e920bd6-241d-44ae-b456-59f84a2d7438 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:19:55.720660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1109/2190] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:19:55 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:19:55.821203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e86979-6dfc-42d5-8103-9d3ab18bc1eb service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:19:55.889386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e86979-6dfc-42d5-8103-9d3ab18bc1eb service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:19:55.893930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fad47c0-14a4-4949-9ca1-44c0df6eae10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:19:55.894553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fad47c0-14a4-4949-9ca1-44c0df6eae10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:55.894828 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fad47c0-14a4-4949-9ca1-44c0df6eae10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:55.895100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fad47c0-14a4-4949-9ca1-44c0df6eae10 admin admin] PATH_INFO: `/v3/users/2b3682aef6f8415f9a5cac2db2f7bff9` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:55.896215 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fad47c0-14a4-4949-9ca1-44c0df6eae10 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b3682aef6f8415f9a5cac2db2f7bff9)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:19:55.898158 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fad47c0-14a4-4949-9ca1-44c0df6eae10 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:19:55.959525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5fad47c0-14a4-4949-9ca1-44c0df6eae10 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b3682aef6f8415f9a5cac2db2f7bff9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:19:55.960180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5fad47c0-14a4-4949-9ca1-44c0df6eae10 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b3682aef6f8415f9a5cac2db2f7bff9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:19:55.960825 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5fad47c0-14a4-4949-9ca1-44c0df6eae10 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b3682aef6f8415f9a5cac2db2f7bff9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:19:55.969165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5fad47c0-14a4-4949-9ca1-44c0df6eae10 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b3682aef6f8415f9a5cac2db2f7bff9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:19:55.981766 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5fad47c0-14a4-4949-9ca1-44c0df6eae10 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b3682aef6f8415f9a5cac2db2f7bff9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:19:55.993551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1082/2191] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:19:55 2022] DELETE /identity/v3/users/2b3682aef6f8415f9a5cac2db2f7bff9 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:19:56.164196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e920bd6-241d-44ae-b456-59f84a2d7438 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:19:56.177908 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e920bd6-241d-44ae-b456-59f84a2d7438 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:19:56.233697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:19:56.234312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:56.234591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:56.234861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b admin admin] PATH_INFO: `/v3/projects/1d5209c6077a4fd592a983d1430f630b` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:56.235956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d5209c6077a4fd592a983d1430f630b)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:19:56.238041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:19:56.283558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:19:56.292425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d5209c6077a4fd592a983d1430f630b', 'request_id': 'req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:19:56.298370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d5209c6077a4fd592a983d1430f630b', 'request_id': 'req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:19:56.305616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d5209c6077a4fd592a983d1430f630b', 'request_id': 'req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:19:56.317147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b admin admin] The token cache is being invalidate because project 1d5209c6077a4fd592a983d1430f630b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:19:56.317445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:19:56.318068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4c9c32ab-85e0-4ff1-b405-58c6e143eb0b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:19:56.319540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1110/2192] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:19:56 2022] DELETE /identity/v3/projects/1d5209c6077a4fd592a983d1430f630b => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:19:56.328108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4f3cd8-00a6-4be1-812e-a3af581f5dbc None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:56.328449 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4f3cd8-00a6-4be1-812e-a3af581f5dbc None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:56.328730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4f3cd8-00a6-4be1-812e-a3af581f5dbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:56.337278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:19:56.349212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-2f4f3cd8-00a6-4be1-812e-a3af581f5dbc None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:19:56.350436 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2f4f3cd8-00a6-4be1-812e-a3af581f5dbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:19:56.407697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1083/2193] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:19:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:19:56.415634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bfdc45-95d5-4d90-8210-0e4a8816653e None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:56.415920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bfdc45-95d5-4d90-8210-0e4a8816653e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:56.416196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bfdc45-95d5-4d90-8210-0e4a8816653e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:56.436889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-88bfdc45-95d5-4d90-8210-0e4a8816653e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:19:56.438555 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-88bfdc45-95d5-4d90-8210-0e4a8816653e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:19:56.447283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1111/2194] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:19:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:19:56.451445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4f3cd8-00a6-4be1-812e-a3af581f5dbc None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:19:56.461009 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4f3cd8-00a6-4be1-812e-a3af581f5dbc None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:19:56.465546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b64fef-3200-4541-9ca4-ed9ff13bdca0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:19:56.466132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b64fef-3200-4541-9ca4-ed9ff13bdca0 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:56.466410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b64fef-3200-4541-9ca4-ed9ff13bdca0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:56.466771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b64fef-3200-4541-9ca4-ed9ff13bdca0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:56.467410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b64fef-3200-4541-9ca4-ed9ff13bdca0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:19:56.467712 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b64fef-3200-4541-9ca4-ed9ff13bdca0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:19:56.469376 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b64fef-3200-4541-9ca4-ed9ff13bdca0 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:19:56.481330 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1084/2195] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:19:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:19:56.485537 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bfdc45-95d5-4d90-8210-0e4a8816653e None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:19:56.495181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bfdc45-95d5-4d90-8210-0e4a8816653e None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:19:56.500109 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4f9ea7-a746-411c-91fc-0cbede9fbadb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:19:56.500737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4f9ea7-a746-411c-91fc-0cbede9fbadb admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:56.501018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4f9ea7-a746-411c-91fc-0cbede9fbadb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:56.501298 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4f9ea7-a746-411c-91fc-0cbede9fbadb admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:56.501939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4f9ea7-a746-411c-91fc-0cbede9fbadb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:19:56.503850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4f9ea7-a746-411c-91fc-0cbede9fbadb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:19:56.530348 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1112/2196] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:19:56 2022] POST /identity/v3/projects => generated 363 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:19:56.534941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b64fef-3200-4541-9ca4-ed9ff13bdca0 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:19:56.545896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b64fef-3200-4541-9ca4-ed9ff13bdca0 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:19:56.615690 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018b57fd-1d91-41fc-9789-3fb237e07afb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:19:56.616392 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018b57fd-1d91-41fc-9789-3fb237e07afb admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:56.616680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018b57fd-1d91-41fc-9789-3fb237e07afb admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:56.617023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018b57fd-1d91-41fc-9789-3fb237e07afb admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:56.617655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018b57fd-1d91-41fc-9789-3fb237e07afb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:19:56.619573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018b57fd-1d91-41fc-9789-3fb237e07afb admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:19:56.642939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1085/2197] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:19:56 2022] POST /identity/v3/users => generated 282 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:19:56.647319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4f9ea7-a746-411c-91fc-0cbede9fbadb admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:19:56.658537 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4f9ea7-a746-411c-91fc-0cbede9fbadb admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:19:56.663177 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9e1113-162a-4cc2-9665-ed9cfa79e100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:19:56.663773 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9e1113-162a-4cc2-9665-ed9cfa79e100 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:56.664052 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9e1113-162a-4cc2-9665-ed9cfa79e100 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:56.664347 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9e1113-162a-4cc2-9665-ed9cfa79e100 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:56.664995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9e1113-162a-4cc2-9665-ed9cfa79e100 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:19:56.666562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9e1113-162a-4cc2-9665-ed9cfa79e100 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:19:56.677748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1113/2198] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:19:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:19:56.681157 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018b57fd-1d91-41fc-9789-3fb237e07afb admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:19:56.691041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018b57fd-1d91-41fc-9789-3fb237e07afb admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:19:56.695618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6598d2e-e6a0-4707-976c-e69e1879dae1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:19:56.696201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6598d2e-e6a0-4707-976c-e69e1879dae1 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:56.696510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6598d2e-e6a0-4707-976c-e69e1879dae1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:56.696783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6598d2e-e6a0-4707-976c-e69e1879dae1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:56.697419 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6598d2e-e6a0-4707-976c-e69e1879dae1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:19:56.699061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6598d2e-e6a0-4707-976c-e69e1879dae1 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:19:56.709524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1086/2199] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:19:56 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:19:56.713448 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9e1113-162a-4cc2-9665-ed9cfa79e100 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:19:56.724117 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9e1113-162a-4cc2-9665-ed9cfa79e100 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:19:56.728789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e5e505-42d5-4584-b7c3-12313d024eab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:19:56.729446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e5e505-42d5-4584-b7c3-12313d024eab admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:56.729720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e5e505-42d5-4584-b7c3-12313d024eab admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:56.729993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e5e505-42d5-4584-b7c3-12313d024eab admin admin] PATH_INFO: `/v3/projects/57f81fca7d7442918cb2a9e79b7d55a6/users/d3e0b0152841456b91bcc072bf69fce1/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:56.744699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e5e505-42d5-4584-b7c3-12313d024eab admin admin] RBAC: Authorizing `identity:create_grant(project_id=57f81fca7d7442918cb2a9e79b7d55a6, user_id=d3e0b0152841456b91bcc072bf69fce1, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:19:56.747887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e5e505-42d5-4584-b7c3-12313d024eab admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:19:56.761236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1114/2200] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:19:56 2022] PUT /identity/v3/projects/57f81fca7d7442918cb2a9e79b7d55a6/users/d3e0b0152841456b91bcc072bf69fce1/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:19:56.772847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b974b8b-0cfa-4986-a4e9-1936eb0f4201 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:56.773133 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b974b8b-0cfa-4986-a4e9-1936eb0f4201 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:56.773405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b974b8b-0cfa-4986-a4e9-1936eb0f4201 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:56.801125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3b974b8b-0cfa-4986-a4e9-1936eb0f4201 None None] MFA Rules not processed for user `d3e0b0152841456b91bcc072bf69fce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:19:56.802245 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3b974b8b-0cfa-4986-a4e9-1936eb0f4201 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:19:56.861467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:19:56.865630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1087/2201] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:19:56 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:19:56.914477 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e5e505-42d5-4584-b7c3-12313d024eab admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:19:56.916351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-18e5e505-42d5-4584-b7c3-12313d024eab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:19:56.973117 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e5e505-42d5-4584-b7c3-12313d024eab admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:19:56.976704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5e95f2-aa6f-43a0-b2fd-1bb8b6825367 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:19:56.977299 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e95f2-aa6f-43a0-b2fd-1bb8b6825367 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:56.977573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e95f2-aa6f-43a0-b2fd-1bb8b6825367 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:56.977843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e95f2-aa6f-43a0-b2fd-1bb8b6825367 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:56.985969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5e95f2-aa6f-43a0-b2fd-1bb8b6825367 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:19:56.987368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5e95f2-aa6f-43a0-b2fd-1bb8b6825367 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:19:56.995850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1115/2202] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:19:56 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:19:58.693031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b974b8b-0cfa-4986-a4e9-1936eb0f4201 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:19:58.702604 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b974b8b-0cfa-4986-a4e9-1936eb0f4201 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:19:58.706190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2d833d-48f4-457c-b291-b97406668a8e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:19:58.706794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2d833d-48f4-457c-b291-b97406668a8e service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:19:58.707070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2d833d-48f4-457c-b291-b97406668a8e service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:19:58.707339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2d833d-48f4-457c-b291-b97406668a8e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:19:58.709100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1f2d833d-48f4-457c-b291-b97406668a8e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:19:58.769539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2d833d-48f4-457c-b291-b97406668a8e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:19:58.771003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2d833d-48f4-457c-b291-b97406668a8e service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:19:58.780133 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1088/2203] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:19:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:20:00.954834 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5e95f2-aa6f-43a0-b2fd-1bb8b6825367 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:20:00.963036 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5e95f2-aa6f-43a0-b2fd-1bb8b6825367 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:20:00.967445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb4f272-92a8-4dcb-b531-04e4d69421f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:20:00.968037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb4f272-92a8-4dcb-b531-04e4d69421f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:20:00.968338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb4f272-92a8-4dcb-b531-04e4d69421f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:20:00.968606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb4f272-92a8-4dcb-b531-04e4d69421f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:20:00.970876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1cb4f272-92a8-4dcb-b531-04e4d69421f6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:20:01.033019 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb4f272-92a8-4dcb-b531-04e4d69421f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:20:01.034472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb4f272-92a8-4dcb-b531-04e4d69421f6 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:20:01.043792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1116/2204] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:20:00 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:20:08.776003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=28 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:20:11.039493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:20:47.497385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2d833d-48f4-457c-b291-b97406668a8e service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:20:47.498833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1f2d833d-48f4-457c-b291-b97406668a8e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:20:47.559438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2d833d-48f4-457c-b291-b97406668a8e service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:20:47.563088 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49592228-8016-45b5-bd95-de26de898b06 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:20:47.563688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49592228-8016-45b5-bd95-de26de898b06 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:20:47.563970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49592228-8016-45b5-bd95-de26de898b06 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:20:47.564243 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49592228-8016-45b5-bd95-de26de898b06 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:20:47.565976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-49592228-8016-45b5-bd95-de26de898b06 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:20:47.575651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49592228-8016-45b5-bd95-de26de898b06 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:20:47.577111 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49592228-8016-45b5-bd95-de26de898b06 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:20:47.584840 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1089/2205] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:20:47 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:20:57.581425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=26 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:21:11.036052 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb4f272-92a8-4dcb-b531-04e4d69421f6 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:21:11.045581 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb4f272-92a8-4dcb-b531-04e4d69421f6 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:21:11.049001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1cb2ce-f720-49e1-8421-eb3bc3c270b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:21:11.049595 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1cb2ce-f720-49e1-8421-eb3bc3c270b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:21:11.049872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1cb2ce-f720-49e1-8421-eb3bc3c270b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:21:11.050144 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1cb2ce-f720-49e1-8421-eb3bc3c270b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:21:11.051884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-fd1cb2ce-f720-49e1-8421-eb3bc3c270b8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:21:11.125171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1cb2ce-f720-49e1-8421-eb3bc3c270b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:21:11.126609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1cb2ce-f720-49e1-8421-eb3bc3c270b8 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:21:11.140766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1117/2206] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:21:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:21:11.822172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49592228-8016-45b5-bd95-de26de898b06 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:21:11.824578 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-49592228-8016-45b5-bd95-de26de898b06 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:21:11.842609 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49592228-8016-45b5-bd95-de26de898b06 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:21:11.850534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c1885e-f3f5-4f1d-ab7e-83cea147a7d0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:21:11.852440 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c1885e-f3f5-4f1d-ab7e-83cea147a7d0 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:21:11.852948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c1885e-f3f5-4f1d-ab7e-83cea147a7d0 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:21:11.853608 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c1885e-f3f5-4f1d-ab7e-83cea147a7d0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:21:11.856039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-69c1885e-f3f5-4f1d-ab7e-83cea147a7d0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:21:11.877270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c1885e-f3f5-4f1d-ab7e-83cea147a7d0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:21:11.879597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c1885e-f3f5-4f1d-ab7e-83cea147a7d0 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:21:11.896639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1090/2207] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:21:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:21:12.866384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1cb2ce-f720-49e1-8421-eb3bc3c270b8 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:21:12.878702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1cb2ce-f720-49e1-8421-eb3bc3c270b8 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:21:12.886404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f86802-aad3-4d4d-98db-7c492c01bc15 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:21:12.887332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f86802-aad3-4d4d-98db-7c492c01bc15 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:21:12.887817 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f86802-aad3-4d4d-98db-7c492c01bc15 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:21:12.888280 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f86802-aad3-4d4d-98db-7c492c01bc15 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:21:12.893680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-39f86802-aad3-4d4d-98db-7c492c01bc15 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:21:12.910629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f86802-aad3-4d4d-98db-7c492c01bc15 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:21:12.912225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f86802-aad3-4d4d-98db-7c492c01bc15 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:21:12.925951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1118/2208] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:21:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:21:15.368213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c1885e-f3f5-4f1d-ab7e-83cea147a7d0 service nova] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:21:15.377766 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c1885e-f3f5-4f1d-ab7e-83cea147a7d0 service nova] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:21:15.381351 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12965ccc-48fe-41da-8239-038bd7166f8d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:21:15.381938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12965ccc-48fe-41da-8239-038bd7166f8d service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:21:15.382215 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12965ccc-48fe-41da-8239-038bd7166f8d service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:21:15.382485 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12965ccc-48fe-41da-8239-038bd7166f8d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:21:15.384171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-12965ccc-48fe-41da-8239-038bd7166f8d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:21:15.441446 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12965ccc-48fe-41da-8239-038bd7166f8d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:21:15.442915 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12965ccc-48fe-41da-8239-038bd7166f8d service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:21:15.452288 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1091/2209] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:21:15 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:21:22.918287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=40 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:21:24.781288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f86802-aad3-4d4d-98db-7c492c01bc15 service nova] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:21:24.782810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-39f86802-aad3-4d4d-98db-7c492c01bc15 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:21:24.796262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f86802-aad3-4d4d-98db-7c492c01bc15 service nova] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:21:24.801750 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7887177f-9ab9-4fe1-9166-bd32f4963732 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:21:24.802342 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7887177f-9ab9-4fe1-9166-bd32f4963732 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:21:24.802621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7887177f-9ab9-4fe1-9166-bd32f4963732 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:21:24.802896 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7887177f-9ab9-4fe1-9166-bd32f4963732 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:21:24.804571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7887177f-9ab9-4fe1-9166-bd32f4963732 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:21:24.871815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7887177f-9ab9-4fe1-9166-bd32f4963732 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:21:24.873418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7887177f-9ab9-4fe1-9166-bd32f4963732 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:21:24.882870 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1119/2210] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:21:24 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:21:25.447723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=34 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:21:34.878666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=28 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:22:26.949042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12965ccc-48fe-41da-8239-038bd7166f8d service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:22:26.957726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12965ccc-48fe-41da-8239-038bd7166f8d service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:22:26.961321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346c16f1-e423-44a5-a64e-ae015cac13e2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:22:26.961902 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346c16f1-e423-44a5-a64e-ae015cac13e2 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:22:26.962178 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346c16f1-e423-44a5-a64e-ae015cac13e2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:22:26.962445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346c16f1-e423-44a5-a64e-ae015cac13e2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:22:26.964153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-346c16f1-e423-44a5-a64e-ae015cac13e2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:22:27.023622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346c16f1-e423-44a5-a64e-ae015cac13e2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:22:27.026216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346c16f1-e423-44a5-a64e-ae015cac13e2 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:22:27.037632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1092/2211] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:22:26 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:22:37.033390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=28 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:23:51.717633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7887177f-9ab9-4fe1-9166-bd32f4963732 service nova] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:23:51.719064 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7887177f-9ab9-4fe1-9166-bd32f4963732 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:23:51.880982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7887177f-9ab9-4fe1-9166-bd32f4963732 service nova] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:23:51.887238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0b12e0-37b8-4d91-9d50-608a2bd60b45 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:23:51.888506 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b12e0-37b8-4d91-9d50-608a2bd60b45 service placement] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:23:51.888790 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b12e0-37b8-4d91-9d50-608a2bd60b45 service placement] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:23:51.889062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b12e0-37b8-4d91-9d50-608a2bd60b45 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:23:51.890832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-fd0b12e0-37b8-4d91-9d50-608a2bd60b45 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:23:51.902385 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0b12e0-37b8-4d91-9d50-608a2bd60b45 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:23:51.904130 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0b12e0-37b8-4d91-9d50-608a2bd60b45 service placement] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:23:51.914045 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1120/2212] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:23:51 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:24:01.909911 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=63 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:24:47.242458 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346c16f1-e423-44a5-a64e-ae015cac13e2 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:47.244136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-346c16f1-e423-44a5-a64e-ae015cac13e2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:47.363333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346c16f1-e423-44a5-a64e-ae015cac13e2 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:47.367932 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336fd38-d726-458f-9cd4-9c0859a04494 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:47.368590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336fd38-d726-458f-9cd4-9c0859a04494 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:47.368868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336fd38-d726-458f-9cd4-9c0859a04494 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:47.369144 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336fd38-d726-458f-9cd4-9c0859a04494 admin admin] PATH_INFO: `/v3/users/6d9e364f2e12418d8d53d8c0dd272b38` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:47.370283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3336fd38-d726-458f-9cd4-9c0859a04494 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d9e364f2e12418d8d53d8c0dd272b38)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:47.372258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3336fd38-d726-458f-9cd4-9c0859a04494 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:47.416678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3336fd38-d726-458f-9cd4-9c0859a04494 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d9e364f2e12418d8d53d8c0dd272b38'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:24:47.417315 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3336fd38-d726-458f-9cd4-9c0859a04494 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d9e364f2e12418d8d53d8c0dd272b38'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:24:47.418551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3336fd38-d726-458f-9cd4-9c0859a04494 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d9e364f2e12418d8d53d8c0dd272b38'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:24:47.425281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3336fd38-d726-458f-9cd4-9c0859a04494 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d9e364f2e12418d8d53d8c0dd272b38'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:24:47.436630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3336fd38-d726-458f-9cd4-9c0859a04494 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d9e364f2e12418d8d53d8c0dd272b38'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:24:47.448816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1093/2213] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:24:47 2022] DELETE /identity/v3/users/6d9e364f2e12418d8d53d8c0dd272b38 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:24:47.453689 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6b170-46bd-4aa8-be54-51e204a91635 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:47.454234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6b170-46bd-4aa8-be54-51e204a91635 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:47.454234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6b170-46bd-4aa8-be54-51e204a91635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:47.482575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-76d6b170-46bd-4aa8-be54-51e204a91635 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:24:47.483783 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-76d6b170-46bd-4aa8-be54-51e204a91635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:47.542572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1121/2214] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:24:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:24:47.552922 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336fd38-d726-458f-9cd4-9c0859a04494 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:47.554359 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3336fd38-d726-458f-9cd4-9c0859a04494 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:47.580750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:24:47.632783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336fd38-d726-458f-9cd4-9c0859a04494 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:47.636474 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5c62f2-49aa-4d4c-8114-c786fdb30d85 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:47.637713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5c62f2-49aa-4d4c-8114-c786fdb30d85 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:47.637995 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5c62f2-49aa-4d4c-8114-c786fdb30d85 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:47.638264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5c62f2-49aa-4d4c-8114-c786fdb30d85 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:47.647202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5c62f2-49aa-4d4c-8114-c786fdb30d85 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:47.648664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5c62f2-49aa-4d4c-8114-c786fdb30d85 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:47.658618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1094/2215] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:24:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:24:47.830942 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d6b170-46bd-4aa8-be54-51e204a91635 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:47.832470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-76d6b170-46bd-4aa8-be54-51e204a91635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:47.845502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d6b170-46bd-4aa8-be54-51e204a91635 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:47.860172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d6993d-6e87-40f8-b04b-0c1895d8a240 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:47.860901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d6993d-6e87-40f8-b04b-0c1895d8a240 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:47.861185 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d6993d-6e87-40f8-b04b-0c1895d8a240 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:47.861458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d6993d-6e87-40f8-b04b-0c1895d8a240 admin admin] PATH_INFO: `/v3/projects/b3720873acbd42708f9ab86b749dae3d` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:47.862731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d6993d-6e87-40f8-b04b-0c1895d8a240 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3720873acbd42708f9ab86b749dae3d)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:47.864856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d6993d-6e87-40f8-b04b-0c1895d8a240 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:47.923436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-11d6993d-6e87-40f8-b04b-0c1895d8a240 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3720873acbd42708f9ab86b749dae3d', 'request_id': 'req-11d6993d-6e87-40f8-b04b-0c1895d8a240'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:24:47.929500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-11d6993d-6e87-40f8-b04b-0c1895d8a240 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3720873acbd42708f9ab86b749dae3d', 'request_id': 'req-11d6993d-6e87-40f8-b04b-0c1895d8a240'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:24:47.938737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-11d6993d-6e87-40f8-b04b-0c1895d8a240 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3720873acbd42708f9ab86b749dae3d', 'request_id': 'req-11d6993d-6e87-40f8-b04b-0c1895d8a240'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:24:47.951255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-11d6993d-6e87-40f8-b04b-0c1895d8a240 admin admin] The token cache is being invalidate because project b3720873acbd42708f9ab86b749dae3d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:24:47.951560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-11d6993d-6e87-40f8-b04b-0c1895d8a240 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:24:47.952215 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-11d6993d-6e87-40f8-b04b-0c1895d8a240 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:24:47.953742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1122/2216] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:24:47 2022] DELETE /identity/v3/projects/b3720873acbd42708f9ab86b749dae3d => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:24:47.960917 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26377e33-a07d-4b93-b9c4-a95a47a3dfac None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:47.961201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26377e33-a07d-4b93-b9c4-a95a47a3dfac None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:47.961470 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26377e33-a07d-4b93-b9c4-a95a47a3dfac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:48.019246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-26377e33-a07d-4b93-b9c4-a95a47a3dfac None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:24:48.020792 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-26377e33-a07d-4b93-b9c4-a95a47a3dfac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:48.100993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1095/2217] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:24:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:24:48.109456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba310e-653f-43a2-b09c-deac6d8361ba None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:48.109809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba310e-653f-43a2-b09c-deac6d8361ba None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:48.110016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba310e-653f-43a2-b09c-deac6d8361ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:48.130473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-90ba310e-653f-43a2-b09c-deac6d8361ba None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:24:48.131644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-90ba310e-653f-43a2-b09c-deac6d8361ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:48.141318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1123/2218] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:24:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:24:48.145363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26377e33-a07d-4b93-b9c4-a95a47a3dfac None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:48.155643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26377e33-a07d-4b93-b9c4-a95a47a3dfac None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:48.160281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa91b3d-b922-4cd0-81eb-34af7cb85db5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:48.160915 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa91b3d-b922-4cd0-81eb-34af7cb85db5 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:48.161191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa91b3d-b922-4cd0-81eb-34af7cb85db5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:48.161463 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa91b3d-b922-4cd0-81eb-34af7cb85db5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:48.162107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa91b3d-b922-4cd0-81eb-34af7cb85db5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:24:48.162404 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa91b3d-b922-4cd0-81eb-34af7cb85db5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:48.164076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa91b3d-b922-4cd0-81eb-34af7cb85db5 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:48.176706 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1096/2219] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:24:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:24:48.180977 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ba310e-653f-43a2-b09c-deac6d8361ba None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:48.190517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ba310e-653f-43a2-b09c-deac6d8361ba None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:48.195314 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3282185b-641a-4e6e-ab62-a6967104077a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:48.195910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3282185b-641a-4e6e-ab62-a6967104077a admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:48.196187 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3282185b-641a-4e6e-ab62-a6967104077a admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:48.196496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3282185b-641a-4e6e-ab62-a6967104077a admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:48.197140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3282185b-641a-4e6e-ab62-a6967104077a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:48.199017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3282185b-641a-4e6e-ab62-a6967104077a admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:48.240181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1124/2220] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:24:48 2022] POST /identity/v3/projects => generated 389 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:24:48.241206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa91b3d-b922-4cd0-81eb-34af7cb85db5 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:48.251393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa91b3d-b922-4cd0-81eb-34af7cb85db5 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:48.318223 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6255dc-59cd-4829-9a74-55969bbfe4d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:48.318832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6255dc-59cd-4829-9a74-55969bbfe4d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:48.319110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6255dc-59cd-4829-9a74-55969bbfe4d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:48.319381 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6255dc-59cd-4829-9a74-55969bbfe4d9 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:48.320016 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6255dc-59cd-4829-9a74-55969bbfe4d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:48.322004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6255dc-59cd-4829-9a74-55969bbfe4d9 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:48.340866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:24:48.346830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1097/2221] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:24:48 2022] POST /identity/v3/users => generated 295 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:24:48.350889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3282185b-641a-4e6e-ab62-a6967104077a admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:48.361718 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3282185b-641a-4e6e-ab62-a6967104077a admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:48.366320 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0e9569-4809-4fff-b28c-ecdd86ad1bfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:48.366910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0e9569-4809-4fff-b28c-ecdd86ad1bfb admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:48.367189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0e9569-4809-4fff-b28c-ecdd86ad1bfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:48.367459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0e9569-4809-4fff-b28c-ecdd86ad1bfb admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:48.368099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0e9569-4809-4fff-b28c-ecdd86ad1bfb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:48.369707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0e9569-4809-4fff-b28c-ecdd86ad1bfb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:48.379411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1125/2222] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:24:48 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:24:48.383195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6255dc-59cd-4829-9a74-55969bbfe4d9 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:48.393540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6255dc-59cd-4829-9a74-55969bbfe4d9 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:48.398113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e59eddc-0568-4819-9894-5d94daa5a7cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:48.398698 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59eddc-0568-4819-9894-5d94daa5a7cd admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:48.398972 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59eddc-0568-4819-9894-5d94daa5a7cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:48.399241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59eddc-0568-4819-9894-5d94daa5a7cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:48.399872 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e59eddc-0568-4819-9894-5d94daa5a7cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:48.401495 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e59eddc-0568-4819-9894-5d94daa5a7cd admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:48.412464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1098/2223] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:24:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:24:48.416530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0e9569-4809-4fff-b28c-ecdd86ad1bfb admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:48.426217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0e9569-4809-4fff-b28c-ecdd86ad1bfb admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:48.430792 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ec1b6f-24a4-4c49-a47e-17bdbc4a5523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:48.431431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec1b6f-24a4-4c49-a47e-17bdbc4a5523 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:48.431690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec1b6f-24a4-4c49-a47e-17bdbc4a5523 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:48.431960 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec1b6f-24a4-4c49-a47e-17bdbc4a5523 admin admin] PATH_INFO: `/v3/projects/d59824afe7634074afb285cf6ff8c48f/users/27735bf5ecfa41a28bef39560b4b6c5e/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:48.446096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ec1b6f-24a4-4c49-a47e-17bdbc4a5523 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d59824afe7634074afb285cf6ff8c48f, user_id=27735bf5ecfa41a28bef39560b4b6c5e, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:48.448756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ec1b6f-24a4-4c49-a47e-17bdbc4a5523 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:48.461472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1126/2224] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:24:48 2022] PUT /identity/v3/projects/d59824afe7634074afb285cf6ff8c48f/users/27735bf5ecfa41a28bef39560b4b6c5e/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:24:48.471586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c23a82-033f-4017-a106-08c25551d23a None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:48.471851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c23a82-033f-4017-a106-08c25551d23a None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:48.472122 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c23a82-033f-4017-a106-08c25551d23a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:48.501397 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-77c23a82-033f-4017-a106-08c25551d23a None None] MFA Rules not processed for user `27735bf5ecfa41a28bef39560b4b6c5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:24:48.502612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-77c23a82-033f-4017-a106-08c25551d23a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:48.564114 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1099/2225] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:24:48 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:24:48.575302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ec1b6f-24a4-4c49-a47e-17bdbc4a5523 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:48.577101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-29ec1b6f-24a4-4c49-a47e-17bdbc4a5523 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:48.583018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:24:48.660249 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ec1b6f-24a4-4c49-a47e-17bdbc4a5523 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:48.664176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f35b902-a59c-485c-8873-5d4635e84b6a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:48.664827 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f35b902-a59c-485c-8873-5d4635e84b6a service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:48.665107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f35b902-a59c-485c-8873-5d4635e84b6a service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:48.665378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f35b902-a59c-485c-8873-5d4635e84b6a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:48.673886 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f35b902-a59c-485c-8873-5d4635e84b6a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:48.675375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f35b902-a59c-485c-8873-5d4635e84b6a service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:48.684733 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1127/2226] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:24:48 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:24:50.229699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c23a82-033f-4017-a106-08c25551d23a None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:50.240418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c23a82-033f-4017-a106-08c25551d23a None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:50.244466 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9480a371-d69e-4aab-aab9-7da38c21a684 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:50.245261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9480a371-d69e-4aab-aab9-7da38c21a684 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:50.248430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9480a371-d69e-4aab-aab9-7da38c21a684 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:50.248812 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9480a371-d69e-4aab-aab9-7da38c21a684 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:50.250706 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9480a371-d69e-4aab-aab9-7da38c21a684 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:50.342763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9480a371-d69e-4aab-aab9-7da38c21a684 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:50.344209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9480a371-d69e-4aab-aab9-7da38c21a684 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:50.364649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1100/2227] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:24:50 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:24:50.650226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f35b902-a59c-485c-8873-5d4635e84b6a service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:50.651658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5f35b902-a59c-485c-8873-5d4635e84b6a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:50.757687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f35b902-a59c-485c-8873-5d4635e84b6a service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:50.763085 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52ca9ba-561c-457e-bd02-e1b81ffbc98e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:50.763706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52ca9ba-561c-457e-bd02-e1b81ffbc98e service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:50.763981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52ca9ba-561c-457e-bd02-e1b81ffbc98e service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:50.764252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52ca9ba-561c-457e-bd02-e1b81ffbc98e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:50.766775 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c52ca9ba-561c-457e-bd02-e1b81ffbc98e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:50.892426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52ca9ba-561c-457e-bd02-e1b81ffbc98e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:50.894240 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52ca9ba-561c-457e-bd02-e1b81ffbc98e service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:50.932520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1128/2228] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:24:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:24:54.100984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95acc571-73fc-4548-b1e4-738d2e776645 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:54.100984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95acc571-73fc-4548-b1e4-738d2e776645 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:54.100984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95acc571-73fc-4548-b1e4-738d2e776645 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:54.140693 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-95acc571-73fc-4548-b1e4-738d2e776645 None None] MFA Rules not processed for user `820f4bb963644499a77fa1f3edd59d08`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:24:54.141914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-95acc571-73fc-4548-b1e4-738d2e776645 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:54.165311 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:24:54.194916 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1101/2229] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:24:54 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:24:54.513523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52ca9ba-561c-457e-bd02-e1b81ffbc98e service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:54.514929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c52ca9ba-561c-457e-bd02-e1b81ffbc98e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:54.520990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:24:54.526410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52ca9ba-561c-457e-bd02-e1b81ffbc98e service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:54.530451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dff2c2e-848e-494e-8b54-dab92c690724 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:54.531044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dff2c2e-848e-494e-8b54-dab92c690724 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:54.531324 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dff2c2e-848e-494e-8b54-dab92c690724 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:54.531597 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dff2c2e-848e-494e-8b54-dab92c690724 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:54.533376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1dff2c2e-848e-494e-8b54-dab92c690724 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:54.542574 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dff2c2e-848e-494e-8b54-dab92c690724 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:54.544534 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dff2c2e-848e-494e-8b54-dab92c690724 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:54.552377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1129/2230] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:24:54 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:24:54.866293 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95acc571-73fc-4548-b1e4-738d2e776645 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:54.867765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-95acc571-73fc-4548-b1e4-738d2e776645 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:54.936550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95acc571-73fc-4548-b1e4-738d2e776645 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:54.941476 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3168c9d4-9dc1-4dc9-81e1-7a9f4b85040e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:54.942077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3168c9d4-9dc1-4dc9-81e1-7a9f4b85040e service placement] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:54.942356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3168c9d4-9dc1-4dc9-81e1-7a9f4b85040e service placement] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:54.942627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3168c9d4-9dc1-4dc9-81e1-7a9f4b85040e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:54.944356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3168c9d4-9dc1-4dc9-81e1-7a9f4b85040e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:54.955567 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3168c9d4-9dc1-4dc9-81e1-7a9f4b85040e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:54.957188 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3168c9d4-9dc1-4dc9-81e1-7a9f4b85040e service placement] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:54.967363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1102/2231] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:24:54 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:24:55.536598 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dff2c2e-848e-494e-8b54-dab92c690724 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:55.546719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dff2c2e-848e-494e-8b54-dab92c690724 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:55.550368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30ad3ee-9656-4807-8dfa-d6903ad9dc52 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:55.551153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30ad3ee-9656-4807-8dfa-d6903ad9dc52 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:55.551534 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30ad3ee-9656-4807-8dfa-d6903ad9dc52 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:55.551805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30ad3ee-9656-4807-8dfa-d6903ad9dc52 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:55.553566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d30ad3ee-9656-4807-8dfa-d6903ad9dc52 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:55.635199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30ad3ee-9656-4807-8dfa-d6903ad9dc52 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:55.637232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30ad3ee-9656-4807-8dfa-d6903ad9dc52 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:55.656153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1130/2232] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:24:55 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:24:57.505109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3168c9d4-9dc1-4dc9-81e1-7a9f4b85040e service placement] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:57.514242 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3168c9d4-9dc1-4dc9-81e1-7a9f4b85040e service placement] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:57.517788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3c81f9-ca50-4738-a9ba-3178ee1b5857 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:57.518376 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c81f9-ca50-4738-a9ba-3178ee1b5857 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:57.518652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c81f9-ca50-4738-a9ba-3178ee1b5857 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:57.518923 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c81f9-ca50-4738-a9ba-3178ee1b5857 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:57.520679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5a3c81f9-ca50-4738-a9ba-3178ee1b5857 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:57.581552 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3c81f9-ca50-4738-a9ba-3178ee1b5857 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:57.582998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3c81f9-ca50-4738-a9ba-3178ee1b5857 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:57.592186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1103/2233] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:24:57 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:24:58.740631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30ad3ee-9656-4807-8dfa-d6903ad9dc52 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:24:58.749073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30ad3ee-9656-4807-8dfa-d6903ad9dc52 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:24:58.752642 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392d6783-98bc-4b69-9c8e-a8e34e1ed892 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:24:58.753217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392d6783-98bc-4b69-9c8e-a8e34e1ed892 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:24:58.753491 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392d6783-98bc-4b69-9c8e-a8e34e1ed892 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:24:58.753761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392d6783-98bc-4b69-9c8e-a8e34e1ed892 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:24:58.755475 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-392d6783-98bc-4b69-9c8e-a8e34e1ed892 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:24:58.834777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392d6783-98bc-4b69-9c8e-a8e34e1ed892 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:24:58.836714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392d6783-98bc-4b69-9c8e-a8e34e1ed892 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:24:58.856418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1131/2234] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:24:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:25:00.585401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3c81f9-ca50-4738-a9ba-3178ee1b5857 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:25:00.586863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5a3c81f9-ca50-4738-a9ba-3178ee1b5857 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:25:00.604028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3c81f9-ca50-4738-a9ba-3178ee1b5857 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:25:00.609368 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4c08ab-eb91-486c-b17b-901e0a63d8c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:25:00.609957 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c08ab-eb91-486c-b17b-901e0a63d8c6 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:25:00.610234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c08ab-eb91-486c-b17b-901e0a63d8c6 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:25:00.610502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c08ab-eb91-486c-b17b-901e0a63d8c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:25:00.612241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-fd4c08ab-eb91-486c-b17b-901e0a63d8c6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:25:00.625128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4c08ab-eb91-486c-b17b-901e0a63d8c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:25:00.626741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4c08ab-eb91-486c-b17b-901e0a63d8c6 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:25:00.645756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1104/2235] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:25:00 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:25:04.201073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392d6783-98bc-4b69-9c8e-a8e34e1ed892 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:25:04.215197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392d6783-98bc-4b69-9c8e-a8e34e1ed892 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:25:04.223352 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f729829d-e528-4b95-b026-608f4f14d3f8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:25:04.224723 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f729829d-e528-4b95-b026-608f4f14d3f8 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:25:04.225169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f729829d-e528-4b95-b026-608f4f14d3f8 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:25:04.225777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f729829d-e528-4b95-b026-608f4f14d3f8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:25:04.228781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f729829d-e528-4b95-b026-608f4f14d3f8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:25:04.251160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f729829d-e528-4b95-b026-608f4f14d3f8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:25:04.252743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f729829d-e528-4b95-b026-608f4f14d3f8 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:25:04.279240 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1132/2236] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:25:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:25:06.850591 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eacb15f-0a95-4faa-8241-d3f815b1d021 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:25:06.851095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eacb15f-0a95-4faa-8241-d3f815b1d021 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:25:06.851270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eacb15f-0a95-4faa-8241-d3f815b1d021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:25:06.879986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-6eacb15f-0a95-4faa-8241-d3f815b1d021 None None] MFA Rules not processed for user `fdb0f345d2e04483b5166e1eb728bf91`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:25:06.881175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6eacb15f-0a95-4faa-8241-d3f815b1d021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:25:06.888671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1105/2237] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:25:06 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:25:06.941271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737829cd-0876-4a24-bfa0-dea4b4878e43 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:25:06.941753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737829cd-0876-4a24-bfa0-dea4b4878e43 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:25:06.941836 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737829cd-0876-4a24-bfa0-dea4b4878e43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:25:06.975486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-737829cd-0876-4a24-bfa0-dea4b4878e43 None None] MFA Rules not processed for user `4a8818fd59954480bcff68b9ba52769a`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:25:06.976751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-737829cd-0876-4a24-bfa0-dea4b4878e43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:25:06.988343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1133/2238] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:25:06 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:25:07.205060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eacb15f-0a95-4faa-8241-d3f815b1d021 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:25:07.216379 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eacb15f-0a95-4faa-8241-d3f815b1d021 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:25:07.219900 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f02724-b917-4075-9db7-0c392385f59f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:25:07.220531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f02724-b917-4075-9db7-0c392385f59f service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:25:07.220986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f02724-b917-4075-9db7-0c392385f59f service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:25:07.221420 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f02724-b917-4075-9db7-0c392385f59f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:25:07.223342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-06f02724-b917-4075-9db7-0c392385f59f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:25:07.235137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f02724-b917-4075-9db7-0c392385f59f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:25:07.236347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f02724-b917-4075-9db7-0c392385f59f service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:25:07.244772 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1106/2239] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:25:07 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:25:16.973208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:25:17.240882 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=94 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:25:52.503025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737829cd-0876-4a24-bfa0-dea4b4878e43 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:25:52.512722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737829cd-0876-4a24-bfa0-dea4b4878e43 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:25:52.516554 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978e93c0-bc11-4dad-9ea4-6dddf01709b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:25:52.517178 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978e93c0-bc11-4dad-9ea4-6dddf01709b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:25:52.517457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978e93c0-bc11-4dad-9ea4-6dddf01709b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:25:52.517733 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978e93c0-bc11-4dad-9ea4-6dddf01709b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:25:52.519427 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-978e93c0-bc11-4dad-9ea4-6dddf01709b8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:25:52.529376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978e93c0-bc11-4dad-9ea4-6dddf01709b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:25:52.530797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978e93c0-bc11-4dad-9ea4-6dddf01709b8 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:25:52.538514 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1134/2240] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:25:52 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:26:02.535079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=6 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:26:14.969984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f02724-b917-4075-9db7-0c392385f59f service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:26:14.971371 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-06f02724-b917-4075-9db7-0c392385f59f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:26:14.983583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f02724-b917-4075-9db7-0c392385f59f service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:26:14.987226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3e618b-cae6-4053-ab55-59a125981605 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:26:14.987813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3e618b-cae6-4053-ab55-59a125981605 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:26:14.988092 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3e618b-cae6-4053-ab55-59a125981605 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:26:14.988393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3e618b-cae6-4053-ab55-59a125981605 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:26:14.990080 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1d3e618b-cae6-4053-ab55-59a125981605 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:26:15.067461 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3e618b-cae6-4053-ab55-59a125981605 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:26:15.068977 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3e618b-cae6-4053-ab55-59a125981605 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:26:15.077752 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1107/2241] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:26:14 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:26:25.051177 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978e93c0-bc11-4dad-9ea4-6dddf01709b8 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:26:25.052692 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-978e93c0-bc11-4dad-9ea4-6dddf01709b8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:26:25.068789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978e93c0-bc11-4dad-9ea4-6dddf01709b8 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:26:25.073659 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=37 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:26:25.074222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730e82aa-9b5a-4157-abb9-fdb038033c5f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:26:25.074841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e82aa-9b5a-4157-abb9-fdb038033c5f service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:26:25.075089 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e82aa-9b5a-4157-abb9-fdb038033c5f service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:26:25.075388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e82aa-9b5a-4157-abb9-fdb038033c5f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:26:25.077110 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-730e82aa-9b5a-4157-abb9-fdb038033c5f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:26:25.154147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730e82aa-9b5a-4157-abb9-fdb038033c5f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:26:25.155738 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730e82aa-9b5a-4157-abb9-fdb038033c5f service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:26:25.179593 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1135/2242] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:26:25 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:26:35.176963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=41 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:28:55.523391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3e618b-cae6-4053-ab55-59a125981605 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:28:55.535250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3e618b-cae6-4053-ab55-59a125981605 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:28:55.540565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8544b34-0be5-4c04-87ee-28e76cac551a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:28:55.541124 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8544b34-0be5-4c04-87ee-28e76cac551a service placement] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:28:55.541402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8544b34-0be5-4c04-87ee-28e76cac551a service placement] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:28:55.541678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8544b34-0be5-4c04-87ee-28e76cac551a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:28:55.544040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a8544b34-0be5-4c04-87ee-28e76cac551a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:28:55.555140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8544b34-0be5-4c04-87ee-28e76cac551a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:28:55.556746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8544b34-0be5-4c04-87ee-28e76cac551a service placement] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:28:55.566334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1108/2243] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:28:55 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:29:05.562230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=6 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:29:34.707074 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730e82aa-9b5a-4157-abb9-fdb038033c5f service nova] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:29:34.716269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730e82aa-9b5a-4157-abb9-fdb038033c5f service nova] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:29:34.719813 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd562349-4632-4d4f-ba8c-d35ce8a4e0eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:29:34.720434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd562349-4632-4d4f-ba8c-d35ce8a4e0eb service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:29:34.720712 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd562349-4632-4d4f-ba8c-d35ce8a4e0eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:29:34.720985 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd562349-4632-4d4f-ba8c-d35ce8a4e0eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:29:34.722683 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-cd562349-4632-4d4f-ba8c-d35ce8a4e0eb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:29:34.736167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd562349-4632-4d4f-ba8c-d35ce8a4e0eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:29:34.737606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd562349-4632-4d4f-ba8c-d35ce8a4e0eb service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:29:34.748837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1136/2244] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:29:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:29:44.743381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=6 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:29:49.103044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8544b34-0be5-4c04-87ee-28e76cac551a service placement] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:29:49.112116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8544b34-0be5-4c04-87ee-28e76cac551a service placement] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:29:49.115638 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8397b920-7363-4029-88b0-056aa22dbfd4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:29:49.116214 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8397b920-7363-4029-88b0-056aa22dbfd4 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:29:49.116524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8397b920-7363-4029-88b0-056aa22dbfd4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:29:49.116792 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8397b920-7363-4029-88b0-056aa22dbfd4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:29:49.125249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8397b920-7363-4029-88b0-056aa22dbfd4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:29:49.126687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8397b920-7363-4029-88b0-056aa22dbfd4 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:29:49.135401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1109/2245] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:29:49 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 33 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:29:50.267875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd562349-4632-4d4f-ba8c-d35ce8a4e0eb service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:29:50.277582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd562349-4632-4d4f-ba8c-d35ce8a4e0eb service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:29:50.281166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6a9c44-3290-41bb-8a50-cb2a5df642a0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:29:50.281744 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6a9c44-3290-41bb-8a50-cb2a5df642a0 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:29:50.282018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6a9c44-3290-41bb-8a50-cb2a5df642a0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:29:50.282285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6a9c44-3290-41bb-8a50-cb2a5df642a0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:29:50.290687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6a9c44-3290-41bb-8a50-cb2a5df642a0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:29:50.292153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6a9c44-3290-41bb-8a50-cb2a5df642a0 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:29:50.300802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1137/2246] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:29:50 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 33 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:29:52.362843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8397b920-7363-4029-88b0-056aa22dbfd4 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:29:52.380529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8397b920-7363-4029-88b0-056aa22dbfd4 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:29:52.380529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94ef37f-05b5-422a-9b57-9a9a7fc4bf6a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:29:52.380529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ef37f-05b5-422a-9b57-9a9a7fc4bf6a service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:29:52.380529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ef37f-05b5-422a-9b57-9a9a7fc4bf6a service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:29:52.381009 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ef37f-05b5-422a-9b57-9a9a7fc4bf6a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:29:52.389930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94ef37f-05b5-422a-9b57-9a9a7fc4bf6a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:29:52.391357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94ef37f-05b5-422a-9b57-9a9a7fc4bf6a service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:29:52.402602 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1110/2247] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:29:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:29:56.701632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6a9c44-3290-41bb-8a50-cb2a5df642a0 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:29:56.711497 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6a9c44-3290-41bb-8a50-cb2a5df642a0 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:29:56.715134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56366f34-b4c2-41ea-b93c-89c6f1175fd6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:29:56.715713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56366f34-b4c2-41ea-b93c-89c6f1175fd6 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:29:56.715989 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56366f34-b4c2-41ea-b93c-89c6f1175fd6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:29:56.716282 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56366f34-b4c2-41ea-b93c-89c6f1175fd6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:29:56.725944 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56366f34-b4c2-41ea-b93c-89c6f1175fd6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:29:56.727737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56366f34-b4c2-41ea-b93c-89c6f1175fd6 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:29:56.737620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1138/2248] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:29:56 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:29:57.716750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94ef37f-05b5-422a-9b57-9a9a7fc4bf6a service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:29:57.726016 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94ef37f-05b5-422a-9b57-9a9a7fc4bf6a service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:29:57.729898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb51773b-cb0c-4945-9279-a1625e8e7f3f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:29:57.730474 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb51773b-cb0c-4945-9279-a1625e8e7f3f service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:29:57.730745 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb51773b-cb0c-4945-9279-a1625e8e7f3f service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:29:57.731009 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb51773b-cb0c-4945-9279-a1625e8e7f3f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:29:57.739240 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb51773b-cb0c-4945-9279-a1625e8e7f3f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:29:57.740689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb51773b-cb0c-4945-9279-a1625e8e7f3f service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:29:57.767742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1111/2249] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:29:57 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:29:59.207368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56366f34-b4c2-41ea-b93c-89c6f1175fd6 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:29:59.216450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56366f34-b4c2-41ea-b93c-89c6f1175fd6 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:29:59.219988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8a0693-986f-45c4-8a00-1268f099c3df service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:29:59.220715 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8a0693-986f-45c4-8a00-1268f099c3df service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:29:59.221009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8a0693-986f-45c4-8a00-1268f099c3df service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:29:59.221269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8a0693-986f-45c4-8a00-1268f099c3df service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:29:59.230481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8a0693-986f-45c4-8a00-1268f099c3df service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:29:59.231879 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8a0693-986f-45c4-8a00-1268f099c3df service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:29:59.240513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1139/2250] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:29:59 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:30:00.762151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb51773b-cb0c-4945-9279-a1625e8e7f3f service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:30:00.770598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb51773b-cb0c-4945-9279-a1625e8e7f3f service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:30:00.774253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e2edc4-d3de-4c0f-850e-27d0f7449a1f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:30:00.774854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e2edc4-d3de-4c0f-850e-27d0f7449a1f service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:30:00.775128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e2edc4-d3de-4c0f-850e-27d0f7449a1f service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:30:00.775395 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e2edc4-d3de-4c0f-850e-27d0f7449a1f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:30:00.784172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e2edc4-d3de-4c0f-850e-27d0f7449a1f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:30:00.785615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e2edc4-d3de-4c0f-850e-27d0f7449a1f service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:30:00.794210 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1112/2251] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:30:00 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 33 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:30:09.236515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=18 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:30:10.790215 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=31 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:30:49.265670 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8a0693-986f-45c4-8a00-1268f099c3df service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:30:49.337181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8a0693-986f-45c4-8a00-1268f099c3df service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:30:49.342429 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87dc5def-ccff-48ed-9a73-8edec4c4cd72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:30:49.343162 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dc5def-ccff-48ed-9a73-8edec4c4cd72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:30:49.343528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dc5def-ccff-48ed-9a73-8edec4c4cd72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:30:49.343815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dc5def-ccff-48ed-9a73-8edec4c4cd72 admin admin] PATH_INFO: `/v3/users/27735bf5ecfa41a28bef39560b4b6c5e` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:30:49.344962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87dc5def-ccff-48ed-9a73-8edec4c4cd72 admin admin] RBAC: Authorizing `identity:delete_user(user_id=27735bf5ecfa41a28bef39560b4b6c5e)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:30:49.346879 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87dc5def-ccff-48ed-9a73-8edec4c4cd72 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:30:49.399233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-87dc5def-ccff-48ed-9a73-8edec4c4cd72 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '27735bf5ecfa41a28bef39560b4b6c5e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:30:49.399918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-87dc5def-ccff-48ed-9a73-8edec4c4cd72 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '27735bf5ecfa41a28bef39560b4b6c5e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:30:49.407815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-87dc5def-ccff-48ed-9a73-8edec4c4cd72 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '27735bf5ecfa41a28bef39560b4b6c5e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:30:49.408530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-87dc5def-ccff-48ed-9a73-8edec4c4cd72 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '27735bf5ecfa41a28bef39560b4b6c5e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:30:49.421841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-87dc5def-ccff-48ed-9a73-8edec4c4cd72 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '27735bf5ecfa41a28bef39560b4b6c5e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:30:49.434870 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1140/2252] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:30:49 2022] DELETE /identity/v3/users/27735bf5ecfa41a28bef39560b4b6c5e => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:30:49.439329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467f113b-9f8f-467f-8288-98182a876bb6 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:30:49.439847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467f113b-9f8f-467f-8288-98182a876bb6 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:30:49.439847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467f113b-9f8f-467f-8288-98182a876bb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:30:49.462620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-467f113b-9f8f-467f-8288-98182a876bb6 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:30:49.463805 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-467f113b-9f8f-467f-8288-98182a876bb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:30:49.533345 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1113/2253] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:30:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:30:49.541951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87dc5def-ccff-48ed-9a73-8edec4c4cd72 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:30:49.543571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-87dc5def-ccff-48ed-9a73-8edec4c4cd72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:30:49.614094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87dc5def-ccff-48ed-9a73-8edec4c4cd72 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:30:49.618192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759ed3e4-520d-46a2-b9f4-fdf6ad6cb509 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:30:49.618878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759ed3e4-520d-46a2-b9f4-fdf6ad6cb509 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:30:49.619341 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759ed3e4-520d-46a2-b9f4-fdf6ad6cb509 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:30:49.619478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759ed3e4-520d-46a2-b9f4-fdf6ad6cb509 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:30:49.633955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759ed3e4-520d-46a2-b9f4-fdf6ad6cb509 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:30:49.636557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759ed3e4-520d-46a2-b9f4-fdf6ad6cb509 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:30:49.648971 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1141/2254] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:30:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:30:49.942613 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467f113b-9f8f-467f-8288-98182a876bb6 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:30:49.944421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-467f113b-9f8f-467f-8288-98182a876bb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:30:49.963887 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467f113b-9f8f-467f-8288-98182a876bb6 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:30:49.971764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:30:49.972507 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:30:49.972858 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:30:49.973205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] PATH_INFO: `/v3/projects/d59824afe7634074afb285cf6ff8c48f` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:30:49.974436 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=d59824afe7634074afb285cf6ff8c48f)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:30:49.977578 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:30:50.059371 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd59824afe7634074afb285cf6ff8c48f', 'request_id': 'req-8ad69367-8024-4765-8f50-584bcb7de7ae'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:30:50.070494 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd59824afe7634074afb285cf6ff8c48f', 'request_id': 'req-8ad69367-8024-4765-8f50-584bcb7de7ae'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:30:50.088339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd59824afe7634074afb285cf6ff8c48f', 'request_id': 'req-8ad69367-8024-4765-8f50-584bcb7de7ae'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:30:50.100933 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] The token cache is being invalidate because project d59824afe7634074afb285cf6ff8c48f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:30:50.101270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:30:50.102097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:30:50.103623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1114/2255] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:30:49 2022] DELETE /identity/v3/projects/d59824afe7634074afb285cf6ff8c48f => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:30:57.437500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759ed3e4-520d-46a2-b9f4-fdf6ad6cb509 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:30:57.445970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759ed3e4-520d-46a2-b9f4-fdf6ad6cb509 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:30:57.465201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:30:57.503799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bbd941-2390-4b87-b8ab-e444eb25db64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:30:57.504423 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bbd941-2390-4b87-b8ab-e444eb25db64 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:30:57.504763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bbd941-2390-4b87-b8ab-e444eb25db64 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:30:57.505040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bbd941-2390-4b87-b8ab-e444eb25db64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:30:57.506780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-61bbd941-2390-4b87-b8ab-e444eb25db64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:30:57.516766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bbd941-2390-4b87-b8ab-e444eb25db64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:30:57.518177 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bbd941-2390-4b87-b8ab-e444eb25db64 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:30:57.526075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1142/2256] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:30:57 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:31:00.097518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:31:07.522461 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=16 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:31:14.676385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:31:14.677844 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:31:14.752190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad69367-8024-4765-8f50-584bcb7de7ae admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:31:14.757743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17160b90-2fc4-4be5-96d9-ad136024a63a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:31:14.758345 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17160b90-2fc4-4be5-96d9-ad136024a63a service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:31:14.758623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17160b90-2fc4-4be5-96d9-ad136024a63a service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:31:14.758897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17160b90-2fc4-4be5-96d9-ad136024a63a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:31:14.760680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-17160b90-2fc4-4be5-96d9-ad136024a63a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:31:14.785304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bbd941-2390-4b87-b8ab-e444eb25db64 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:31:14.786688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-61bbd941-2390-4b87-b8ab-e444eb25db64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:31:14.833258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17160b90-2fc4-4be5-96d9-ad136024a63a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:31:14.835470 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17160b90-2fc4-4be5-96d9-ad136024a63a service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:31:14.850377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bbd941-2390-4b87-b8ab-e444eb25db64 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:31:14.854305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8985b03-4bb8-441e-888f-db3b682af4ab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:31:14.854974 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8985b03-4bb8-441e-888f-db3b682af4ab service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:31:14.855258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8985b03-4bb8-441e-888f-db3b682af4ab service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:31:14.855532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8985b03-4bb8-441e-888f-db3b682af4ab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:31:14.859318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1115/2257] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:31:14 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:31:14.864371 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8985b03-4bb8-441e-888f-db3b682af4ab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:31:14.866296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8985b03-4bb8-441e-888f-db3b682af4ab service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:31:14.876274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1143/2258] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:31:14 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:31:24.855973 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=60 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:31:24.871308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=26 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:31:24.972518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17160b90-2fc4-4be5-96d9-ad136024a63a service nova] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:31:24.973946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-17160b90-2fc4-4be5-96d9-ad136024a63a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:31:24.986229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17160b90-2fc4-4be5-96d9-ad136024a63a service nova] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:31:24.989806 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955f6909-46f9-492a-b6b8-bf048fef3771 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:31:24.990430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955f6909-46f9-492a-b6b8-bf048fef3771 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:31:24.990741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955f6909-46f9-492a-b6b8-bf048fef3771 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:31:24.991016 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955f6909-46f9-492a-b6b8-bf048fef3771 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:31:24.992827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-955f6909-46f9-492a-b6b8-bf048fef3771 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:31:25.050257 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955f6909-46f9-492a-b6b8-bf048fef3771 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:31:25.051694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955f6909-46f9-492a-b6b8-bf048fef3771 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:31:25.061252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1116/2259] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:31:24 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:31:35.056557 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=28 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:32:39.268739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8985b03-4bb8-441e-888f-db3b682af4ab service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:32:39.293870 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8985b03-4bb8-441e-888f-db3b682af4ab service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:32:39.301086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861a8c13-9267-4001-8b0a-fd6d691d8b99 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:32:39.301820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861a8c13-9267-4001-8b0a-fd6d691d8b99 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:32:39.302170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861a8c13-9267-4001-8b0a-fd6d691d8b99 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:32:39.302515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861a8c13-9267-4001-8b0a-fd6d691d8b99 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:32:39.304787 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-861a8c13-9267-4001-8b0a-fd6d691d8b99 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:32:39.321089 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861a8c13-9267-4001-8b0a-fd6d691d8b99 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:32:39.323045 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861a8c13-9267-4001-8b0a-fd6d691d8b99 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:32:39.349239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1144/2260] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:32:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:32:49.333945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=6 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:33:13.457748 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ffca15-8736-4793-a49b-873942794a7a None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:33:13.458343 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ffca15-8736-4793-a49b-873942794a7a None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:33:13.458343 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ffca15-8736-4793-a49b-873942794a7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:33:13.477780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-f3ffca15-8736-4793-a49b-873942794a7a None None] MFA Rules not processed for user `4a8818fd59954480bcff68b9ba52769a`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:33:13.478980 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f3ffca15-8736-4793-a49b-873942794a7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:33:13.537669 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1117/2261] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:33:13 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:33:13.854378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861a8c13-9267-4001-8b0a-fd6d691d8b99 service nova] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:33:13.855685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-861a8c13-9267-4001-8b0a-fd6d691d8b99 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:33:13.866814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861a8c13-9267-4001-8b0a-fd6d691d8b99 service nova] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:33:13.870468 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5c57d7-5f75-4417-8e96-992c19e17d7f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:33:13.871055 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5c57d7-5f75-4417-8e96-992c19e17d7f service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:33:13.871332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5c57d7-5f75-4417-8e96-992c19e17d7f service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:33:13.871604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5c57d7-5f75-4417-8e96-992c19e17d7f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:33:13.873327 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6e5c57d7-5f75-4417-8e96-992c19e17d7f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:33:13.882995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5c57d7-5f75-4417-8e96-992c19e17d7f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:33:13.884485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5c57d7-5f75-4417-8e96-992c19e17d7f service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:33:13.893640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1145/2262] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:33:13 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:33:14.218055 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ffca15-8736-4793-a49b-873942794a7a None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:33:14.220499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f3ffca15-8736-4793-a49b-873942794a7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:33:14.313386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ffca15-8736-4793-a49b-873942794a7a None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:33:14.318867 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c69e44-2598-4910-ab55-984a48810957 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:33:14.319554 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c69e44-2598-4910-ab55-984a48810957 service placement] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:33:14.319926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c69e44-2598-4910-ab55-984a48810957 service placement] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:33:14.320716 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c69e44-2598-4910-ab55-984a48810957 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:33:14.325032 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d6c69e44-2598-4910-ab55-984a48810957 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:33:14.345272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c69e44-2598-4910-ab55-984a48810957 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:33:14.346960 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c69e44-2598-4910-ab55-984a48810957 service placement] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:33:14.358257 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1118/2263] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:33:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:33:17.265556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5c57d7-5f75-4417-8e96-992c19e17d7f service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:33:17.280502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5c57d7-5f75-4417-8e96-992c19e17d7f service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:33:17.286777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91fa406-5d92-40c0-b394-f4e42edcbc90 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:33:17.287852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91fa406-5d92-40c0-b394-f4e42edcbc90 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:33:17.288398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91fa406-5d92-40c0-b394-f4e42edcbc90 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:33:17.288755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91fa406-5d92-40c0-b394-f4e42edcbc90 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:33:17.290660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d91fa406-5d92-40c0-b394-f4e42edcbc90 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:33:17.304430 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91fa406-5d92-40c0-b394-f4e42edcbc90 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:33:17.306017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91fa406-5d92-40c0-b394-f4e42edcbc90 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:33:17.320651 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1146/2264] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:33:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:33:22.614478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c69e44-2598-4910-ab55-984a48810957 service placement] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:33:22.615883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d6c69e44-2598-4910-ab55-984a48810957 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:33:22.631734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c69e44-2598-4910-ab55-984a48810957 service placement] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:33:22.637180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261cbca0-08fc-41c5-a4db-44def59a8dff service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:33:22.637891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261cbca0-08fc-41c5-a4db-44def59a8dff service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:33:22.638175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261cbca0-08fc-41c5-a4db-44def59a8dff service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:33:22.638445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261cbca0-08fc-41c5-a4db-44def59a8dff service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:33:22.640165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-261cbca0-08fc-41c5-a4db-44def59a8dff service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:33:22.652870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261cbca0-08fc-41c5-a4db-44def59a8dff service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:33:22.654457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261cbca0-08fc-41c5-a4db-44def59a8dff service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:33:22.664959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1119/2265] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:33:22 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:33:27.315652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=12 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:33:32.660674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=63 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:34:03.789566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91fa406-5d92-40c0-b394-f4e42edcbc90 service nova] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:34:03.831029 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91fa406-5d92-40c0-b394-f4e42edcbc90 service nova] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:34:03.835985 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8794d691-add0-4e08-8638-5e2fc81910cb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:34:03.836607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8794d691-add0-4e08-8638-5e2fc81910cb service placement] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:34:03.836880 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8794d691-add0-4e08-8638-5e2fc81910cb service placement] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:34:03.837148 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8794d691-add0-4e08-8638-5e2fc81910cb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:34:03.838844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8794d691-add0-4e08-8638-5e2fc81910cb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:34:03.863631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8794d691-add0-4e08-8638-5e2fc81910cb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:34:03.865218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8794d691-add0-4e08-8638-5e2fc81910cb service placement] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:34:03.874708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1147/2266] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:34:03 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:34:13.871090 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=24 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:34:35.007172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261cbca0-08fc-41c5-a4db-44def59a8dff service nova] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:34:35.023096 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261cbca0-08fc-41c5-a4db-44def59a8dff service nova] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:34:35.026679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21980c8-6b34-466b-b499-b355dbd4b2f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:34:35.027264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21980c8-6b34-466b-b499-b355dbd4b2f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:34:35.027540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21980c8-6b34-466b-b499-b355dbd4b2f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:34:35.027811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21980c8-6b34-466b-b499-b355dbd4b2f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:34:35.030051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b21980c8-6b34-466b-b499-b355dbd4b2f6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:34:35.042964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21980c8-6b34-466b-b499-b355dbd4b2f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:34:35.045037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21980c8-6b34-466b-b499-b355dbd4b2f6 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:34:35.055486 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1120/2267] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:34:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:34:45.050862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=9 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:34:52.615375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8794d691-add0-4e08-8638-5e2fc81910cb service placement] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:34:52.616801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8794d691-add0-4e08-8638-5e2fc81910cb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:34:52.660871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8794d691-add0-4e08-8638-5e2fc81910cb service placement] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:34:52.664461 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056cfba0-8ebb-49ec-91b3-1b58afa89c6c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:34:52.665052 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056cfba0-8ebb-49ec-91b3-1b58afa89c6c service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:34:52.665331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056cfba0-8ebb-49ec-91b3-1b58afa89c6c service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:34:52.665609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056cfba0-8ebb-49ec-91b3-1b58afa89c6c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:34:52.667299 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-056cfba0-8ebb-49ec-91b3-1b58afa89c6c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:34:52.713006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056cfba0-8ebb-49ec-91b3-1b58afa89c6c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:34:52.714439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056cfba0-8ebb-49ec-91b3-1b58afa89c6c service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:34:52.735402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1148/2268] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:34:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:35:02.732393 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=39 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:35:14.949773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21980c8-6b34-466b-b499-b355dbd4b2f6 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:35:14.972836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21980c8-6b34-466b-b499-b355dbd4b2f6 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:35:14.976693 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf660a2-e7a1-4668-8826-42a24ef4c033 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:35:14.977297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf660a2-e7a1-4668-8826-42a24ef4c033 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:35:14.977574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf660a2-e7a1-4668-8826-42a24ef4c033 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:35:14.978096 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf660a2-e7a1-4668-8826-42a24ef4c033 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:35:14.979839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-ebf660a2-e7a1-4668-8826-42a24ef4c033 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:35:15.061932 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf660a2-e7a1-4668-8826-42a24ef4c033 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:35:15.064029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf660a2-e7a1-4668-8826-42a24ef4c033 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:35:15.083418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1121/2269] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:35:14 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:35:15.267231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056cfba0-8ebb-49ec-91b3-1b58afa89c6c service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:35:15.276323 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056cfba0-8ebb-49ec-91b3-1b58afa89c6c service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:35:15.279799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f364379-e3b2-42ee-b415-5d379c03e836 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:35:15.280426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f364379-e3b2-42ee-b415-5d379c03e836 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:35:15.280707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f364379-e3b2-42ee-b415-5d379c03e836 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:35:15.280978 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f364379-e3b2-42ee-b415-5d379c03e836 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:35:15.282687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6f364379-e3b2-42ee-b415-5d379c03e836 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:35:15.362472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f364379-e3b2-42ee-b415-5d379c03e836 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:35:15.363828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f364379-e3b2-42ee-b415-5d379c03e836 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:35:15.382203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1149/2270] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:35:15 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:35:15.423938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf660a2-e7a1-4668-8826-42a24ef4c033 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:35:15.432287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf660a2-e7a1-4668-8826-42a24ef4c033 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:35:15.436093 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fe2e1f-66bb-4d9d-ad4d-d5b59055632a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:35:15.436750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fe2e1f-66bb-4d9d-ad4d-d5b59055632a service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:35:15.437036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fe2e1f-66bb-4d9d-ad4d-d5b59055632a service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:35:15.437311 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fe2e1f-66bb-4d9d-ad4d-d5b59055632a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:35:15.439658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e4fe2e1f-66bb-4d9d-ad4d-d5b59055632a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:35:15.514089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fe2e1f-66bb-4d9d-ad4d-d5b59055632a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:35:15.515536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fe2e1f-66bb-4d9d-ad4d-d5b59055632a service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:35:15.524595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1122/2271] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:35:15 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:35:15.568643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f364379-e3b2-42ee-b415-5d379c03e836 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:35:15.577110 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f364379-e3b2-42ee-b415-5d379c03e836 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:35:15.580802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4a4416-e231-4e7e-bb59-9f6e3280ff51 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:35:15.581394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4a4416-e231-4e7e-bb59-9f6e3280ff51 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:35:15.581676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4a4416-e231-4e7e-bb59-9f6e3280ff51 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:35:15.581951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4a4416-e231-4e7e-bb59-9f6e3280ff51 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:35:15.583666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5d4a4416-e231-4e7e-bb59-9f6e3280ff51 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:35:15.663601 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4a4416-e231-4e7e-bb59-9f6e3280ff51 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:35:15.664985 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4a4416-e231-4e7e-bb59-9f6e3280ff51 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:35:15.683351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1150/2272] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:35:15 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:35:15.865042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fe2e1f-66bb-4d9d-ad4d-d5b59055632a service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:35:15.873644 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fe2e1f-66bb-4d9d-ad4d-d5b59055632a service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:35:15.877388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb344279-028e-4b72-9f26-4be01527460d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:35:15.877976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb344279-028e-4b72-9f26-4be01527460d service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:35:15.878257 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb344279-028e-4b72-9f26-4be01527460d service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:35:15.878529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb344279-028e-4b72-9f26-4be01527460d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:35:15.880226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-eb344279-028e-4b72-9f26-4be01527460d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:35:15.915960 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb344279-028e-4b72-9f26-4be01527460d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:35:15.917373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb344279-028e-4b72-9f26-4be01527460d service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:35:15.935571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1123/2273] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:35:15 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:35:25.680918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=82 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:35:25.933050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=99 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:36:04.432158 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4a4416-e231-4e7e-bb59-9f6e3280ff51 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:36:04.441616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4a4416-e231-4e7e-bb59-9f6e3280ff51 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:36:04.445116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca15b6e-4799-45cf-8e34-e24be042fd76 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:36:04.445708 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca15b6e-4799-45cf-8e34-e24be042fd76 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:36:04.445981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca15b6e-4799-45cf-8e34-e24be042fd76 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:36:04.446252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca15b6e-4799-45cf-8e34-e24be042fd76 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:36:04.453762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca15b6e-4799-45cf-8e34-e24be042fd76 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:36:04.455169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca15b6e-4799-45cf-8e34-e24be042fd76 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:36:04.462995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1151/2274] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:36:04 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 31 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:36:14.459658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=6 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:36:26.044367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb344279-028e-4b72-9f26-4be01527460d service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:36:26.057349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb344279-028e-4b72-9f26-4be01527460d service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:36:26.062803 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1586cd-e5f6-4bec-9c5e-eb434efbc2c4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:36:26.063388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1586cd-e5f6-4bec-9c5e-eb434efbc2c4 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:36:26.063664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1586cd-e5f6-4bec-9c5e-eb434efbc2c4 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:36:26.063934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1586cd-e5f6-4bec-9c5e-eb434efbc2c4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:36:26.071589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1586cd-e5f6-4bec-9c5e-eb434efbc2c4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:36:26.073193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1586cd-e5f6-4bec-9c5e-eb434efbc2c4 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:36:26.099641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1124/2275] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:36:26 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:36:36.097038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=13 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:37:16.159747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca15b6e-4799-45cf-8e34-e24be042fd76 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:16.169250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca15b6e-4799-45cf-8e34-e24be042fd76 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:16.172797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5425ff19-bae4-4048-964c-9044616325e9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:16.173372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5425ff19-bae4-4048-964c-9044616325e9 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:16.173643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5425ff19-bae4-4048-964c-9044616325e9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:16.173907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5425ff19-bae4-4048-964c-9044616325e9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:16.185274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5425ff19-bae4-4048-964c-9044616325e9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:16.185274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5425ff19-bae4-4048-964c-9044616325e9 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:16.193639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1152/2276] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:37:16 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:37:21.269208 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1586cd-e5f6-4bec-9c5e-eb434efbc2c4 service nova] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:21.278645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1586cd-e5f6-4bec-9c5e-eb434efbc2c4 service nova] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:21.282521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af56bf2c-ae23-466c-9865-e93468989e43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:21.283269 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af56bf2c-ae23-466c-9865-e93468989e43 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:21.283558 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af56bf2c-ae23-466c-9865-e93468989e43 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:21.283862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af56bf2c-ae23-466c-9865-e93468989e43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:21.286242 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-af56bf2c-ae23-466c-9865-e93468989e43 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:37:21.372738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af56bf2c-ae23-466c-9865-e93468989e43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:21.374221 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af56bf2c-ae23-466c-9865-e93468989e43 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:21.392771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1125/2277] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:37:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:37:24.625509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f50a5-2710-4079-b2ea-390d91838aff None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:24.626336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f50a5-2710-4079-b2ea-390d91838aff None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:24.626847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f50a5-2710-4079-b2ea-390d91838aff None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:24.628566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1153/2278] 10.222.0.26 () {54 vars in 921 bytes} [Wed Sep 28 12:37:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:37:24.658570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391b244c-083e-4628-a73d-5e5810b5f43d None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:24.659028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391b244c-083e-4628-a73d-5e5810b5f43d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:24.659446 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391b244c-083e-4628-a73d-5e5810b5f43d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:24.702509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66f50a5-2710-4079-b2ea-390d91838aff None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:24.704386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f66f50a5-2710-4079-b2ea-390d91838aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:37:24.726339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66f50a5-2710-4079-b2ea-390d91838aff None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:24.739391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-391b244c-083e-4628-a73d-5e5810b5f43d None None] MFA Rules not processed for user `3758ea0f26264340ad2f154c0111af21`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:37:24.740889 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-391b244c-083e-4628-a73d-5e5810b5f43d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:37:24.754869 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aeb610-9f00-4818-b3de-771d3da893dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:24.756097 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aeb610-9f00-4818-b3de-771d3da893dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:24.756639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aeb610-9f00-4818-b3de-771d3da893dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:24.757256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aeb610-9f00-4818-b3de-771d3da893dd admin admin] PATH_INFO: `/v3/users/956c7f92a4a747dba49e3f3a527bba74` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:24.759329 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2aeb610-9f00-4818-b3de-771d3da893dd admin admin] RBAC: Authorizing `identity:delete_user(user_id=956c7f92a4a747dba49e3f3a527bba74)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:24.761101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1126/2279] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Sep 28 12:37:24 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:37:24.762062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2aeb610-9f00-4818-b3de-771d3da893dd admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:24.775539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391b244c-083e-4628-a73d-5e5810b5f43d None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:24.777124 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-391b244c-083e-4628-a73d-5e5810b5f43d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:37:24.793934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391b244c-083e-4628-a73d-5e5810b5f43d None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:24.800989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a60108-e5ad-45a5-8619-ec1d54ce702d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:24.801645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a60108-e5ad-45a5-8619-ec1d54ce702d service placement] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:24.801924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a60108-e5ad-45a5-8619-ec1d54ce702d service placement] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:24.802194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a60108-e5ad-45a5-8619-ec1d54ce702d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:24.811730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a60108-e5ad-45a5-8619-ec1d54ce702d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:24.813364 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a60108-e5ad-45a5-8619-ec1d54ce702d service placement] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:24.818038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a2aeb610-9f00-4818-b3de-771d3da893dd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '956c7f92a4a747dba49e3f3a527bba74'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:37:24.818806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a2aeb610-9f00-4818-b3de-771d3da893dd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '956c7f92a4a747dba49e3f3a527bba74'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:37:24.824843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1127/2280] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:37:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:37:24.826647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a2aeb610-9f00-4818-b3de-771d3da893dd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '956c7f92a4a747dba49e3f3a527bba74'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:37:24.827369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a2aeb610-9f00-4818-b3de-771d3da893dd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '956c7f92a4a747dba49e3f3a527bba74'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:37:24.845983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a2aeb610-9f00-4818-b3de-771d3da893dd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '956c7f92a4a747dba49e3f3a527bba74'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:37:24.859951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1154/2281] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:37:24 2022] DELETE /identity/v3/users/956c7f92a4a747dba49e3f3a527bba74 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:37:25.031966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a60108-e5ad-45a5-8619-ec1d54ce702d service placement] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:25.043075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a60108-e5ad-45a5-8619-ec1d54ce702d service placement] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:25.099348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19606eb-c44b-4976-a124-3226aa753d32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:25.099892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19606eb-c44b-4976-a124-3226aa753d32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:25.100173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19606eb-c44b-4976-a124-3226aa753d32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:25.100483 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19606eb-c44b-4976-a124-3226aa753d32 admin admin] PATH_INFO: `/v3/projects/47157b4286ac4ed5a24ba31bafd206a2` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:25.101622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19606eb-c44b-4976-a124-3226aa753d32 admin admin] RBAC: Authorizing `identity:delete_project(project_id=47157b4286ac4ed5a24ba31bafd206a2)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:25.103591 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19606eb-c44b-4976-a124-3226aa753d32 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:25.120155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:37:25.148809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a19606eb-c44b-4976-a124-3226aa753d32 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47157b4286ac4ed5a24ba31bafd206a2', 'request_id': 'req-a19606eb-c44b-4976-a124-3226aa753d32'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:37:25.154140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a19606eb-c44b-4976-a124-3226aa753d32 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47157b4286ac4ed5a24ba31bafd206a2', 'request_id': 'req-a19606eb-c44b-4976-a124-3226aa753d32'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:37:25.160833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a19606eb-c44b-4976-a124-3226aa753d32 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47157b4286ac4ed5a24ba31bafd206a2', 'request_id': 'req-a19606eb-c44b-4976-a124-3226aa753d32'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:37:25.171627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a19606eb-c44b-4976-a124-3226aa753d32 admin admin] The token cache is being invalidate because project 47157b4286ac4ed5a24ba31bafd206a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:37:25.171914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a19606eb-c44b-4976-a124-3226aa753d32 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:37:25.172560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a19606eb-c44b-4976-a124-3226aa753d32 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:37:25.174111 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1128/2282] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:37:25 2022] DELETE /identity/v3/projects/47157b4286ac4ed5a24ba31bafd206a2 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:37:25.180611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37061697-9fe4-44fc-be92-a5d0d2666a5c None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:25.180903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37061697-9fe4-44fc-be92-a5d0d2666a5c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:25.181185 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37061697-9fe4-44fc-be92-a5d0d2666a5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:25.227652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-37061697-9fe4-44fc-be92-a5d0d2666a5c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:37:25.229079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-37061697-9fe4-44fc-be92-a5d0d2666a5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:37:25.288879 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1155/2283] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:37:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:37:25.296771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304d6b93-3a66-423c-a205-92c66adddc6d None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:25.297056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304d6b93-3a66-423c-a205-92c66adddc6d None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:25.297327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304d6b93-3a66-423c-a205-92c66adddc6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:25.317636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-304d6b93-3a66-423c-a205-92c66adddc6d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:37:25.318825 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-304d6b93-3a66-423c-a205-92c66adddc6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:37:25.326528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1129/2284] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:37:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:37:25.330603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37061697-9fe4-44fc-be92-a5d0d2666a5c None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:25.339974 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37061697-9fe4-44fc-be92-a5d0d2666a5c None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:25.344459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7638e296-97d9-4dff-a9d6-6d1db2e4a9ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:25.345175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7638e296-97d9-4dff-a9d6-6d1db2e4a9ba admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:25.345440 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7638e296-97d9-4dff-a9d6-6d1db2e4a9ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:25.345707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7638e296-97d9-4dff-a9d6-6d1db2e4a9ba admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:25.346351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7638e296-97d9-4dff-a9d6-6d1db2e4a9ba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:37:25.346650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7638e296-97d9-4dff-a9d6-6d1db2e4a9ba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:25.348235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7638e296-97d9-4dff-a9d6-6d1db2e4a9ba admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:25.359664 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1156/2285] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:37:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:37:25.364228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304d6b93-3a66-423c-a205-92c66adddc6d None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:25.373559 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304d6b93-3a66-423c-a205-92c66adddc6d None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:25.378297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2daf6a7-3c34-49dc-a4ce-c516236e21eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:25.379517 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2daf6a7-3c34-49dc-a4ce-c516236e21eb admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:25.379795 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2daf6a7-3c34-49dc-a4ce-c516236e21eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:25.380068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2daf6a7-3c34-49dc-a4ce-c516236e21eb admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:25.380734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2daf6a7-3c34-49dc-a4ce-c516236e21eb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:25.382634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2daf6a7-3c34-49dc-a4ce-c516236e21eb admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:25.418209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1130/2286] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:37:25 2022] POST /identity/v3/projects => generated 371 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:37:25.422355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7638e296-97d9-4dff-a9d6-6d1db2e4a9ba admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:25.432860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7638e296-97d9-4dff-a9d6-6d1db2e4a9ba admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:25.452627 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:37:25.491732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146f38eb-aa41-4709-98fa-76116014aebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:25.492250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146f38eb-aa41-4709-98fa-76116014aebd admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:25.492562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146f38eb-aa41-4709-98fa-76116014aebd admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:25.492837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146f38eb-aa41-4709-98fa-76116014aebd admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:25.493493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146f38eb-aa41-4709-98fa-76116014aebd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:25.495557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146f38eb-aa41-4709-98fa-76116014aebd admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:25.516422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1157/2287] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:37:25 2022] POST /identity/v3/users => generated 286 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:37:25.520607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2daf6a7-3c34-49dc-a4ce-c516236e21eb admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:25.530866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2daf6a7-3c34-49dc-a4ce-c516236e21eb admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:25.535952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0220799-1f9f-4f1c-a6a9-29bf812a4c97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:25.536578 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0220799-1f9f-4f1c-a6a9-29bf812a4c97 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:25.536856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0220799-1f9f-4f1c-a6a9-29bf812a4c97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:25.537123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0220799-1f9f-4f1c-a6a9-29bf812a4c97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:25.537906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0220799-1f9f-4f1c-a6a9-29bf812a4c97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:25.539506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0220799-1f9f-4f1c-a6a9-29bf812a4c97 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:25.549158 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1131/2288] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:37:25 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:37:25.553848 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146f38eb-aa41-4709-98fa-76116014aebd admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:25.566334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146f38eb-aa41-4709-98fa-76116014aebd admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:25.571535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62458eb-85b0-4ee4-87d0-8b0299cd2840 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:25.572127 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62458eb-85b0-4ee4-87d0-8b0299cd2840 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:25.572443 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62458eb-85b0-4ee4-87d0-8b0299cd2840 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:25.572719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62458eb-85b0-4ee4-87d0-8b0299cd2840 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:25.573368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62458eb-85b0-4ee4-87d0-8b0299cd2840 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:25.574945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62458eb-85b0-4ee4-87d0-8b0299cd2840 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:25.586806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1158/2289] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:37:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:37:25.591467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0220799-1f9f-4f1c-a6a9-29bf812a4c97 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:25.602726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0220799-1f9f-4f1c-a6a9-29bf812a4c97 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:25.607827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88292c2b-e898-4b19-b98e-6f09004b2d62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:25.608500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88292c2b-e898-4b19-b98e-6f09004b2d62 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:25.608778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88292c2b-e898-4b19-b98e-6f09004b2d62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:25.609048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88292c2b-e898-4b19-b98e-6f09004b2d62 admin admin] PATH_INFO: `/v3/projects/b1dfa5697a574320b09b8c135be222e5/users/078303aeca5f4f6ea7dacf0ebcd39238/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:25.624632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88292c2b-e898-4b19-b98e-6f09004b2d62 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1dfa5697a574320b09b8c135be222e5, user_id=078303aeca5f4f6ea7dacf0ebcd39238, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:25.627273 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88292c2b-e898-4b19-b98e-6f09004b2d62 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:25.639688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1132/2290] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:37:25 2022] PUT /identity/v3/projects/b1dfa5697a574320b09b8c135be222e5/users/078303aeca5f4f6ea7dacf0ebcd39238/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:37:25.650094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19ebd0f-8be4-403c-82aa-a7b7cbcf8d49 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:25.650599 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19ebd0f-8be4-403c-82aa-a7b7cbcf8d49 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:25.650707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19ebd0f-8be4-403c-82aa-a7b7cbcf8d49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:25.679114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-f19ebd0f-8be4-403c-82aa-a7b7cbcf8d49 None None] MFA Rules not processed for user `078303aeca5f4f6ea7dacf0ebcd39238`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:37:25.680344 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f19ebd0f-8be4-403c-82aa-a7b7cbcf8d49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:37:25.742877 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1159/2291] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:37:25 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:37:25.747228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88292c2b-e898-4b19-b98e-6f09004b2d62 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:25.810248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88292c2b-e898-4b19-b98e-6f09004b2d62 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:25.814950 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9af31a8-e1e7-44ad-b158-a7c95c23092f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:25.815549 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9af31a8-e1e7-44ad-b158-a7c95c23092f admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:25.815827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9af31a8-e1e7-44ad-b158-a7c95c23092f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:25.816103 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9af31a8-e1e7-44ad-b158-a7c95c23092f admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:25.816774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9af31a8-e1e7-44ad-b158-a7c95c23092f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:25.818697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9af31a8-e1e7-44ad-b158-a7c95c23092f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:25.846630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1133/2292] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:37:25 2022] POST /identity/v3/projects => generated 371 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:37:25.851384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19ebd0f-8be4-403c-82aa-a7b7cbcf8d49 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:25.862247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19ebd0f-8be4-403c-82aa-a7b7cbcf8d49 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:25.919626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a55f0a-e5e7-4c43-969e-9466c8927853 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:25.920246 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a55f0a-e5e7-4c43-969e-9466c8927853 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:25.920593 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a55f0a-e5e7-4c43-969e-9466c8927853 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:25.920875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a55f0a-e5e7-4c43-969e-9466c8927853 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:25.921525 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a55f0a-e5e7-4c43-969e-9466c8927853 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:25.923510 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a55f0a-e5e7-4c43-969e-9466c8927853 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:25.950988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1160/2293] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:37:25 2022] POST /identity/v3/users => generated 292 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:37:25.955088 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9af31a8-e1e7-44ad-b158-a7c95c23092f admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:25.967154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9af31a8-e1e7-44ad-b158-a7c95c23092f admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:25.972013 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b726ae-a6b9-4f0f-a1cb-9f0749e75af1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:25.973274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b726ae-a6b9-4f0f-a1cb-9f0749e75af1 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:25.973606 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b726ae-a6b9-4f0f-a1cb-9f0749e75af1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:25.973879 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b726ae-a6b9-4f0f-a1cb-9f0749e75af1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:25.974518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b726ae-a6b9-4f0f-a1cb-9f0749e75af1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:25.976113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b726ae-a6b9-4f0f-a1cb-9f0749e75af1 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:25.988083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1134/2294] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:37:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:37:25.992340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a55f0a-e5e7-4c43-969e-9466c8927853 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:26.002584 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a55f0a-e5e7-4c43-969e-9466c8927853 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:26.007212 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35aa996-3689-420f-b003-6b691b73629d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:26.007846 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35aa996-3689-420f-b003-6b691b73629d admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:26.008121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35aa996-3689-420f-b003-6b691b73629d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:26.008540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35aa996-3689-420f-b003-6b691b73629d admin admin] PATH_INFO: `/v3/projects/e71c7904570b4e2689ac789184673f0d/users/eef0c202fa2d4d3a8cde1cc227a164b4/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:26.023653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35aa996-3689-420f-b003-6b691b73629d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e71c7904570b4e2689ac789184673f0d, user_id=eef0c202fa2d4d3a8cde1cc227a164b4, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:26.026325 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35aa996-3689-420f-b003-6b691b73629d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:26.034355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:37:26.040968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1161/2295] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:37:25 2022] PUT /identity/v3/projects/e71c7904570b4e2689ac789184673f0d/users/eef0c202fa2d4d3a8cde1cc227a164b4/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:37:26.045065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b726ae-a6b9-4f0f-a1cb-9f0749e75af1 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:26.055258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b726ae-a6b9-4f0f-a1cb-9f0749e75af1 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:26.058812 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:37:26.112819 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7be6d7-551b-49be-9d59-a24f0d1baa36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:26.113968 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7be6d7-551b-49be-9d59-a24f0d1baa36 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:26.114255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7be6d7-551b-49be-9d59-a24f0d1baa36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:26.114531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7be6d7-551b-49be-9d59-a24f0d1baa36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:26.115175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7be6d7-551b-49be-9d59-a24f0d1baa36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:26.116799 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7be6d7-551b-49be-9d59-a24f0d1baa36 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:26.126137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1135/2296] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:37:26 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:37:26.130806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35aa996-3689-420f-b003-6b691b73629d admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:37:26.141352 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35aa996-3689-420f-b003-6b691b73629d admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:37:26.146004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccb5696-81a4-4712-bab7-ed82487b714e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:37:26.146641 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccb5696-81a4-4712-bab7-ed82487b714e admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:26.146916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccb5696-81a4-4712-bab7-ed82487b714e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:26.147186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccb5696-81a4-4712-bab7-ed82487b714e admin admin] PATH_INFO: `/v3/projects/e71c7904570b4e2689ac789184673f0d/users/eef0c202fa2d4d3a8cde1cc227a164b4/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:26.149094 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccb5696-81a4-4712-bab7-ed82487b714e admin admin] RBAC: Authorizing `identity:create_grant(project_id=e71c7904570b4e2689ac789184673f0d, user_id=eef0c202fa2d4d3a8cde1cc227a164b4, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:37:26.151689 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccb5696-81a4-4712-bab7-ed82487b714e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:37:26.162544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1162/2297] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:37:26 2022] PUT /identity/v3/projects/e71c7904570b4e2689ac789184673f0d/users/eef0c202fa2d4d3a8cde1cc227a164b4/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:37:26.173896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eb9587-f02e-4ce2-9438-dc6c49f818ce None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:37:26.174231 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eb9587-f02e-4ce2-9438-dc6c49f818ce None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:37:26.174535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eb9587-f02e-4ce2-9438-dc6c49f818ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:37:26.202319 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-15eb9587-f02e-4ce2-9438-dc6c49f818ce None None] MFA Rules not processed for user `eef0c202fa2d4d3a8cde1cc227a164b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:37:26.203586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-15eb9587-f02e-4ce2-9438-dc6c49f818ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:37:26.270199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1136/2298] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:37:26 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:37:36.158104 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=6,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:37:36.266642 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=60 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:39:06.662629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccb5696-81a4-4712-bab7-ed82487b714e admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:39:06.664336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7ccb5696-81a4-4712-bab7-ed82487b714e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:39:06.732624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccb5696-81a4-4712-bab7-ed82487b714e admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:39:06.737596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360d761f-a282-4d81-99b1-b2cb4e6c1806 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:39:06.738136 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360d761f-a282-4d81-99b1-b2cb4e6c1806 service placement] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:39:06.738418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360d761f-a282-4d81-99b1-b2cb4e6c1806 service placement] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:39:06.738692 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360d761f-a282-4d81-99b1-b2cb4e6c1806 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:39:06.740487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-360d761f-a282-4d81-99b1-b2cb4e6c1806 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:39:06.751794 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360d761f-a282-4d81-99b1-b2cb4e6c1806 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:39:06.753396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360d761f-a282-4d81-99b1-b2cb4e6c1806 service placement] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:39:06.763676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1163/2299] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:39:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:39:14.349254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5866558a-1341-4029-a413-6d5d9a64af00 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:39:14.349789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5866558a-1341-4029-a413-6d5d9a64af00 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:39:14.349903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5866558a-1341-4029-a413-6d5d9a64af00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:39:14.401017 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-5866558a-1341-4029-a413-6d5d9a64af00 None None] MFA Rules not processed for user `b20a6fb4e7a546058f5fa314c9a59299`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:39:14.402232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5866558a-1341-4029-a413-6d5d9a64af00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:39:14.464892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1137/2300] 10.222.0.26 () {58 vars in 967 bytes} [Wed Sep 28 12:39:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:39:14.511875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360d761f-a282-4d81-99b1-b2cb4e6c1806 service placement] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:39:14.513280 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-360d761f-a282-4d81-99b1-b2cb4e6c1806 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:39:14.599256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360d761f-a282-4d81-99b1-b2cb4e6c1806 service placement] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:39:14.604969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90050550-2656-4e7a-a0d1-89bdd1ad76e3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e663aad79bb8429aa223564a15332ee2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:39:14.605557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90050550-2656-4e7a-a0d1-89bdd1ad76e3 service swift] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:39:14.605832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90050550-2656-4e7a-a0d1-89bdd1ad76e3 service swift] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:39:14.606099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90050550-2656-4e7a-a0d1-89bdd1ad76e3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:39:14.613851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90050550-2656-4e7a-a0d1-89bdd1ad76e3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:39:14.616003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90050550-2656-4e7a-a0d1-89bdd1ad76e3 service swift] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:39:14.623744 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1164/2301] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Sep 28 12:39:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 28 12:39:15.183342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5866558a-1341-4029-a413-6d5d9a64af00 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:39:15.184828 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5866558a-1341-4029-a413-6d5d9a64af00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:39:15.241230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5866558a-1341-4029-a413-6d5d9a64af00 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:39:15.244813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedaf7b5-c940-493e-a971-082bac22aea5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:39:15.245413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedaf7b5-c940-493e-a971-082bac22aea5 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:39:15.245688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedaf7b5-c940-493e-a971-082bac22aea5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:39:15.245955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedaf7b5-c940-493e-a971-082bac22aea5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:39:15.247658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-fedaf7b5-c940-493e-a971-082bac22aea5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:39:15.307529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedaf7b5-c940-493e-a971-082bac22aea5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:39:15.308986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedaf7b5-c940-493e-a971-082bac22aea5 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:39:15.318204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1138/2302] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:39:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:39:16.574806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90050550-2656-4e7a-a0d1-89bdd1ad76e3 service swift] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:39:16.576222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-90050550-2656-4e7a-a0d1-89bdd1ad76e3 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:39:16.647062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:39:16.651702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90050550-2656-4e7a-a0d1-89bdd1ad76e3 service swift] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:39:16.680486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e20810-17e4-4526-81f9-af6aecea8582 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:39:16.682116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e20810-17e4-4526-81f9-af6aecea8582 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:39:16.682938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e20810-17e4-4526-81f9-af6aecea8582 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:39:16.683716 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e20810-17e4-4526-81f9-af6aecea8582 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:39:16.704096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e20810-17e4-4526-81f9-af6aecea8582 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:39:16.707268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e20810-17e4-4526-81f9-af6aecea8582 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:39:16.720748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1165/2303] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:39:16 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:39:18.267479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedaf7b5-c940-493e-a971-082bac22aea5 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:39:18.268986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-fedaf7b5-c940-493e-a971-082bac22aea5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:39:18.285114 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedaf7b5-c940-493e-a971-082bac22aea5 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:39:18.290647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1427a1f-993a-4d05-8820-279a327790c9 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:39:18.291354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1427a1f-993a-4d05-8820-279a327790c9 service placement] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:39:18.291732 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1427a1f-993a-4d05-8820-279a327790c9 service placement] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:39:18.292138 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1427a1f-993a-4d05-8820-279a327790c9 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:39:18.294027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e1427a1f-993a-4d05-8820-279a327790c9 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:39:18.306340 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1427a1f-993a-4d05-8820-279a327790c9 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:39:18.308027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1427a1f-993a-4d05-8820-279a327790c9 service placement] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:39:18.319939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1139/2304] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:39:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:39:21.674347 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e20810-17e4-4526-81f9-af6aecea8582 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:39:21.675808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-12e20810-17e4-4526-81f9-af6aecea8582 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:39:21.690805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e20810-17e4-4526-81f9-af6aecea8582 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:39:21.695912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc8e3a6-fa6d-4d87-9e34-8b56bb27448b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:39:21.696559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc8e3a6-fa6d-4d87-9e34-8b56bb27448b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:39:21.696841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc8e3a6-fa6d-4d87-9e34-8b56bb27448b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:39:21.697114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc8e3a6-fa6d-4d87-9e34-8b56bb27448b admin admin] PATH_INFO: `/v3/users/4226c61349b440c3a502a7cbc592d883` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:39:21.711286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc8e3a6-fa6d-4d87-9e34-8b56bb27448b admin admin] RBAC: Authorizing `identity:delete_user(user_id=4226c61349b440c3a502a7cbc592d883)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:39:21.713261 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc8e3a6-fa6d-4d87-9e34-8b56bb27448b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:39:21.757476 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-abc8e3a6-fa6d-4d87-9e34-8b56bb27448b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4226c61349b440c3a502a7cbc592d883'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:21.758043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-abc8e3a6-fa6d-4d87-9e34-8b56bb27448b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4226c61349b440c3a502a7cbc592d883'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:21.764900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-abc8e3a6-fa6d-4d87-9e34-8b56bb27448b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4226c61349b440c3a502a7cbc592d883'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:21.765546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-abc8e3a6-fa6d-4d87-9e34-8b56bb27448b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4226c61349b440c3a502a7cbc592d883'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:21.778256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-abc8e3a6-fa6d-4d87-9e34-8b56bb27448b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4226c61349b440c3a502a7cbc592d883'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:21.790830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1166/2305] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:39:21 2022] DELETE /identity/v3/users/4226c61349b440c3a502a7cbc592d883 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:39:21.963541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1427a1f-993a-4d05-8820-279a327790c9 service placement] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:39:21.983373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1427a1f-993a-4d05-8820-279a327790c9 service placement] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:39:21.991388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:39:22.051200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:39:22.051828 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:39:22.052108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:39:22.052431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] PATH_INFO: `/v3/projects/e271418048c1466e948bf84de04fc05f` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:39:22.054059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] RBAC: Authorizing `identity:delete_project(project_id=e271418048c1466e948bf84de04fc05f)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:39:22.056018 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:39:22.102789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e271418048c1466e948bf84de04fc05f', 'request_id': 'req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.108510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e271418048c1466e948bf84de04fc05f', 'request_id': 'req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.116500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e271418048c1466e948bf84de04fc05f', 'request_id': 'req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.127395 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] The token cache is being invalidate because project e271418048c1466e948bf84de04fc05f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:39:22.127680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.128298 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.129801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1140/2306] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:39:21 2022] DELETE /identity/v3/projects/e271418048c1466e948bf84de04fc05f => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:39:22.134385 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc8e3a6-fa6d-4d87-9e34-8b56bb27448b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:39:22.135767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-abc8e3a6-fa6d-4d87-9e34-8b56bb27448b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:39:22.148372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc8e3a6-fa6d-4d87-9e34-8b56bb27448b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:39:22.203475 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06ff532-f1a3-48d8-b92b-976d24006031 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:39:22.204097 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06ff532-f1a3-48d8-b92b-976d24006031 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:39:22.204406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06ff532-f1a3-48d8-b92b-976d24006031 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:39:22.204680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06ff532-f1a3-48d8-b92b-976d24006031 admin admin] PATH_INFO: `/v3/users/ffe6fdd4cbc64f7291975cc8957ba17b` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:39:22.217391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06ff532-f1a3-48d8-b92b-976d24006031 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ffe6fdd4cbc64f7291975cc8957ba17b)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:39:22.219300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06ff532-f1a3-48d8-b92b-976d24006031 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:39:22.261180 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:39:22.264044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d06ff532-f1a3-48d8-b92b-976d24006031 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ffe6fdd4cbc64f7291975cc8957ba17b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.264779 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d06ff532-f1a3-48d8-b92b-976d24006031 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ffe6fdd4cbc64f7291975cc8957ba17b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.273210 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d06ff532-f1a3-48d8-b92b-976d24006031 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ffe6fdd4cbc64f7291975cc8957ba17b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.273829 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d06ff532-f1a3-48d8-b92b-976d24006031 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ffe6fdd4cbc64f7291975cc8957ba17b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.285515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d06ff532-f1a3-48d8-b92b-976d24006031 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ffe6fdd4cbc64f7291975cc8957ba17b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.297376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1167/2307] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:39:22 2022] DELETE /identity/v3/users/ffe6fdd4cbc64f7291975cc8957ba17b => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:39:22.467488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:39:22.468978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:39:22.490950 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15146dc-fe03-4f9c-b07a-d4bd60ab915e admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:39:22.547671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:39:22.548301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:39:22.548609 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:39:22.548883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] PATH_INFO: `/v3/projects/e9bbda2115b847e6ab8ee5f544fc823a` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:39:22.560458 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9bbda2115b847e6ab8ee5f544fc823a)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:39:22.562403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:39:22.594887 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:39:22.609380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9bbda2115b847e6ab8ee5f544fc823a', 'request_id': 'req-69aaf004-4b0f-47a8-9890-8868fa7005f7'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.614840 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9bbda2115b847e6ab8ee5f544fc823a', 'request_id': 'req-69aaf004-4b0f-47a8-9890-8868fa7005f7'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.622247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9bbda2115b847e6ab8ee5f544fc823a', 'request_id': 'req-69aaf004-4b0f-47a8-9890-8868fa7005f7'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.633276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] The token cache is being invalidate because project e9bbda2115b847e6ab8ee5f544fc823a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:39:22.633562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.634187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:39:22.635640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1141/2308] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:39:22 2022] DELETE /identity/v3/projects/e9bbda2115b847e6ab8ee5f544fc823a => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:39:32.293884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:DELETE=4,keystone:SELECT=13,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:39:32.630346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=12,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:41:09.066013 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06ff532-f1a3-48d8-b92b-976d24006031 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:09.067442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d06ff532-f1a3-48d8-b92b-976d24006031 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:09.123351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06ff532-f1a3-48d8-b92b-976d24006031 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:09.170206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a14498-86e2-484c-8e3b-a2b956ec73af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:09.170798 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a14498-86e2-484c-8e3b-a2b956ec73af service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:09.171074 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a14498-86e2-484c-8e3b-a2b956ec73af service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:09.171342 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a14498-86e2-484c-8e3b-a2b956ec73af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:09.173108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-10a14498-86e2-484c-8e3b-a2b956ec73af service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:09.181878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a14498-86e2-484c-8e3b-a2b956ec73af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:09.183277 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a14498-86e2-484c-8e3b-a2b956ec73af service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:09.190939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1168/2309] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:41:09 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:41:16.887457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:16.888920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:16.908987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a14498-86e2-484c-8e3b-a2b956ec73af service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:16.910442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-10a14498-86e2-484c-8e3b-a2b956ec73af service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:16.967309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69aaf004-4b0f-47a8-9890-8868fa7005f7 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:16.971160 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a187bba9-24fb-4f4b-bc4d-fc05da9f2ebb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:16.971778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a187bba9-24fb-4f4b-bc4d-fc05da9f2ebb service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:16.972062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a187bba9-24fb-4f4b-bc4d-fc05da9f2ebb service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:16.972354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a187bba9-24fb-4f4b-bc4d-fc05da9f2ebb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:16.977148 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a187bba9-24fb-4f4b-bc4d-fc05da9f2ebb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:16.981981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a14498-86e2-484c-8e3b-a2b956ec73af service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:16.986021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb1e180-ce8e-49b4-bf8d-386ec3936b89 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:16.986769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb1e180-ce8e-49b4-bf8d-386ec3936b89 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:16.987073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb1e180-ce8e-49b4-bf8d-386ec3936b89 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:16.987358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb1e180-ce8e-49b4-bf8d-386ec3936b89 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:16.989232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-dfb1e180-ce8e-49b4-bf8d-386ec3936b89 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:17.048713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a187bba9-24fb-4f4b-bc4d-fc05da9f2ebb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:17.049379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb1e180-ce8e-49b4-bf8d-386ec3936b89 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:17.050168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a187bba9-24fb-4f4b-bc4d-fc05da9f2ebb service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:17.050968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb1e180-ce8e-49b4-bf8d-386ec3936b89 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:17.060300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1169/2310] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:41:16 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:41:17.069132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1142/2311] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:41:16 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:41:17.112273 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb1e180-ce8e-49b4-bf8d-386ec3936b89 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:17.122894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb1e180-ce8e-49b4-bf8d-386ec3936b89 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:17.126697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aadf3ea9-1e6f-4528-8e33-63081a0ec275 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:17.127250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadf3ea9-1e6f-4528-8e33-63081a0ec275 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:17.127580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadf3ea9-1e6f-4528-8e33-63081a0ec275 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:17.127799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadf3ea9-1e6f-4528-8e33-63081a0ec275 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:17.129567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-aadf3ea9-1e6f-4528-8e33-63081a0ec275 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:17.139806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:41:17.151270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a187bba9-24fb-4f4b-bc4d-fc05da9f2ebb service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:17.159857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a187bba9-24fb-4f4b-bc4d-fc05da9f2ebb service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:17.164616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9012fed-148b-4c63-b5b6-1bcf028a12f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:17.165201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9012fed-148b-4c63-b5b6-1bcf028a12f4 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:17.165472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9012fed-148b-4c63-b5b6-1bcf028a12f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:17.165742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9012fed-148b-4c63-b5b6-1bcf028a12f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:17.167464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a9012fed-148b-4c63-b5b6-1bcf028a12f4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:17.191760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aadf3ea9-1e6f-4528-8e33-63081a0ec275 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:17.193491 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aadf3ea9-1e6f-4528-8e33-63081a0ec275 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:17.202833 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1170/2312] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:41:17 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:41:17.227023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9012fed-148b-4c63-b5b6-1bcf028a12f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:17.228500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9012fed-148b-4c63-b5b6-1bcf028a12f4 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:17.239093 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1143/2313] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:41:17 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:41:17.525580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aadf3ea9-1e6f-4528-8e33-63081a0ec275 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:17.527198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-aadf3ea9-1e6f-4528-8e33-63081a0ec275 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:17.573640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9012fed-148b-4c63-b5b6-1bcf028a12f4 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:17.575248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a9012fed-148b-4c63-b5b6-1bcf028a12f4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:17.605171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aadf3ea9-1e6f-4528-8e33-63081a0ec275 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:17.610636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e78c2e-3282-444a-9ac0-358c2280c1a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:17.611257 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e78c2e-3282-444a-9ac0-358c2280c1a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:17.611531 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e78c2e-3282-444a-9ac0-358c2280c1a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:17.611801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e78c2e-3282-444a-9ac0-358c2280c1a6 admin admin] PATH_INFO: `/v3/users/d3e0b0152841456b91bcc072bf69fce1` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:17.612919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e78c2e-3282-444a-9ac0-358c2280c1a6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d3e0b0152841456b91bcc072bf69fce1)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:17.614844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e78c2e-3282-444a-9ac0-358c2280c1a6 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:17.629437 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:41:17.655976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9012fed-148b-4c63-b5b6-1bcf028a12f4 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:17.661621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5049a116-00c7-4e98-bc8f-61b9bb67a687 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:17.662244 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5049a116-00c7-4e98-bc8f-61b9bb67a687 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:17.662518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5049a116-00c7-4e98-bc8f-61b9bb67a687 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:17.662789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5049a116-00c7-4e98-bc8f-61b9bb67a687 admin admin] PATH_INFO: `/v3/users/078303aeca5f4f6ea7dacf0ebcd39238` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:17.664671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5049a116-00c7-4e98-bc8f-61b9bb67a687 admin admin] RBAC: Authorizing `identity:delete_user(user_id=078303aeca5f4f6ea7dacf0ebcd39238)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:17.666584 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5049a116-00c7-4e98-bc8f-61b9bb67a687 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:17.667930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-21e78c2e-3282-444a-9ac0-358c2280c1a6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd3e0b0152841456b91bcc072bf69fce1'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:17.668687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-21e78c2e-3282-444a-9ac0-358c2280c1a6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd3e0b0152841456b91bcc072bf69fce1'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:17.675548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-21e78c2e-3282-444a-9ac0-358c2280c1a6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd3e0b0152841456b91bcc072bf69fce1'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:17.676185 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-21e78c2e-3282-444a-9ac0-358c2280c1a6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd3e0b0152841456b91bcc072bf69fce1'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:17.689660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-21e78c2e-3282-444a-9ac0-358c2280c1a6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd3e0b0152841456b91bcc072bf69fce1'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:17.704059 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1171/2314] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:41:17 2022] DELETE /identity/v3/users/d3e0b0152841456b91bcc072bf69fce1 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:41:17.708442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7165853a-01ba-4d0a-a6e1-22d50a025778 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:17.708734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7165853a-01ba-4d0a-a6e1-22d50a025778 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:17.709015 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7165853a-01ba-4d0a-a6e1-22d50a025778 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:17.716701 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5049a116-00c7-4e98-bc8f-61b9bb67a687 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '078303aeca5f4f6ea7dacf0ebcd39238'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:17.717358 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5049a116-00c7-4e98-bc8f-61b9bb67a687 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '078303aeca5f4f6ea7dacf0ebcd39238'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:17.717983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5049a116-00c7-4e98-bc8f-61b9bb67a687 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '078303aeca5f4f6ea7dacf0ebcd39238'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:17.725042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5049a116-00c7-4e98-bc8f-61b9bb67a687 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '078303aeca5f4f6ea7dacf0ebcd39238'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:17.728296 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-7165853a-01ba-4d0a-a6e1-22d50a025778 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:41:17.729516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7165853a-01ba-4d0a-a6e1-22d50a025778 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:17.737227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5049a116-00c7-4e98-bc8f-61b9bb67a687 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '078303aeca5f4f6ea7dacf0ebcd39238'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:17.751186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1144/2315] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:41:17 2022] DELETE /identity/v3/users/078303aeca5f4f6ea7dacf0ebcd39238 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:41:17.755314 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:41:17.755620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffc5b05-4f3e-4986-baf1-e240c30e384b None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:17.755957 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffc5b05-4f3e-4986-baf1-e240c30e384b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:17.756285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffc5b05-4f3e-4986-baf1-e240c30e384b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:17.779931 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-1ffc5b05-4f3e-4986-baf1-e240c30e384b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:41:17.781149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1ffc5b05-4f3e-4986-baf1-e240c30e384b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:17.792861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1172/2316] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:41:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:41:17.802745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7165853a-01ba-4d0a-a6e1-22d50a025778 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:17.804074 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7165853a-01ba-4d0a-a6e1-22d50a025778 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:17.841448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1145/2317] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:41:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:41:17.852794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffc5b05-4f3e-4986-baf1-e240c30e384b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:17.854126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1ffc5b05-4f3e-4986-baf1-e240c30e384b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:17.860167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7165853a-01ba-4d0a-a6e1-22d50a025778 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:17.914106 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffc5b05-4f3e-4986-baf1-e240c30e384b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:17.915678 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0958f43b-51d1-4adf-9ac9-613dfba91a52 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:17.916567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0958f43b-51d1-4adf-9ac9-613dfba91a52 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:17.916872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0958f43b-51d1-4adf-9ac9-613dfba91a52 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:17.917868 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0958f43b-51d1-4adf-9ac9-613dfba91a52 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:17.917998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc422da-cb94-41fa-b5c1-6d0f5e940b23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:17.918550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc422da-cb94-41fa-b5c1-6d0f5e940b23 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:17.918828 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc422da-cb94-41fa-b5c1-6d0f5e940b23 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:17.919097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc422da-cb94-41fa-b5c1-6d0f5e940b23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:17.928892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc422da-cb94-41fa-b5c1-6d0f5e940b23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:17.929175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0958f43b-51d1-4adf-9ac9-613dfba91a52 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:17.930384 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc422da-cb94-41fa-b5c1-6d0f5e940b23 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:17.930609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0958f43b-51d1-4adf-9ac9-613dfba91a52 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:17.934356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:41:17.941111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1173/2318] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:41:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:41:17.942525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1146/2319] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:41:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:41:18.228513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0958f43b-51d1-4adf-9ac9-613dfba91a52 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:18.230413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0958f43b-51d1-4adf-9ac9-613dfba91a52 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:18.254691 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0958f43b-51d1-4adf-9ac9-613dfba91a52 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:18.261984 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:18.262933 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:18.263290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:18.263661 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] PATH_INFO: `/v3/projects/b1dfa5697a574320b09b8c135be222e5` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:18.265160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=b1dfa5697a574320b09b8c135be222e5)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:18.267703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:18.345523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b1dfa5697a574320b09b8c135be222e5', 'request_id': 'req-e9bcd9c3-846d-4693-809c-04ba88b310ce'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.355716 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b1dfa5697a574320b09b8c135be222e5', 'request_id': 'req-e9bcd9c3-846d-4693-809c-04ba88b310ce'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.370208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b1dfa5697a574320b09b8c135be222e5', 'request_id': 'req-e9bcd9c3-846d-4693-809c-04ba88b310ce'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.397425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] The token cache is being invalidate because project b1dfa5697a574320b09b8c135be222e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:41:18.398069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.399647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.402551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1174/2320] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:41:18 2022] DELETE /identity/v3/projects/b1dfa5697a574320b09b8c135be222e5 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:41:18.406211 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc422da-cb94-41fa-b5c1-6d0f5e940b23 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:18.422322 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc422da-cb94-41fa-b5c1-6d0f5e940b23 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:18.446347 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:18.447896 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:18.481778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:41:18.520513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b54981f-d6ff-41f2-8eb1-6bc1a0dde51f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:18.521383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b54981f-d6ff-41f2-8eb1-6bc1a0dde51f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:18.521667 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b54981f-d6ff-41f2-8eb1-6bc1a0dde51f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:18.521941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b54981f-d6ff-41f2-8eb1-6bc1a0dde51f admin admin] PATH_INFO: `/v3/users/eef0c202fa2d4d3a8cde1cc227a164b4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:18.523153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b54981f-d6ff-41f2-8eb1-6bc1a0dde51f admin admin] RBAC: Authorizing `identity:delete_user(user_id=eef0c202fa2d4d3a8cde1cc227a164b4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:18.525148 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b54981f-d6ff-41f2-8eb1-6bc1a0dde51f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:18.568742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bcd9c3-846d-4693-809c-04ba88b310ce admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:18.575514 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:18.576215 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:18.576594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:18.576949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] PATH_INFO: `/v3/projects/57f81fca7d7442918cb2a9e79b7d55a6` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:18.578306 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] RBAC: Authorizing `identity:delete_project(project_id=57f81fca7d7442918cb2a9e79b7d55a6)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:18.580262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:18.587434 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6b54981f-d6ff-41f2-8eb1-6bc1a0dde51f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eef0c202fa2d4d3a8cde1cc227a164b4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.588149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6b54981f-d6ff-41f2-8eb1-6bc1a0dde51f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eef0c202fa2d4d3a8cde1cc227a164b4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.589034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6b54981f-d6ff-41f2-8eb1-6bc1a0dde51f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eef0c202fa2d4d3a8cde1cc227a164b4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.596662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6b54981f-d6ff-41f2-8eb1-6bc1a0dde51f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eef0c202fa2d4d3a8cde1cc227a164b4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.610654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6b54981f-d6ff-41f2-8eb1-6bc1a0dde51f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eef0c202fa2d4d3a8cde1cc227a164b4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.640730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1147/2321] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:41:18 2022] DELETE /identity/v3/users/eef0c202fa2d4d3a8cde1cc227a164b4 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:41:18.642199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b54981f-d6ff-41f2-8eb1-6bc1a0dde51f admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:18.651770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6b54981f-d6ff-41f2-8eb1-6bc1a0dde51f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:18.674887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '57f81fca7d7442918cb2a9e79b7d55a6', 'request_id': 'req-f8e0cd77-e034-4307-89f4-62dc25545d08'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.686558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '57f81fca7d7442918cb2a9e79b7d55a6', 'request_id': 'req-f8e0cd77-e034-4307-89f4-62dc25545d08'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.697436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '57f81fca7d7442918cb2a9e79b7d55a6', 'request_id': 'req-f8e0cd77-e034-4307-89f4-62dc25545d08'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.712492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] The token cache is being invalidate because project 57f81fca7d7442918cb2a9e79b7d55a6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:41:18.712910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.713826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:18.720350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1175/2322] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:41:18 2022] DELETE /identity/v3/projects/57f81fca7d7442918cb2a9e79b7d55a6 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:41:18.750414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b54981f-d6ff-41f2-8eb1-6bc1a0dde51f admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:18.769156 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:41:18.855398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d4fd72-0a05-4974-bfa4-c250e527314c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:18.856501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d4fd72-0a05-4974-bfa4-c250e527314c service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:18.856899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d4fd72-0a05-4974-bfa4-c250e527314c service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:18.857281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d4fd72-0a05-4974-bfa4-c250e527314c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:18.859300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-33d4fd72-0a05-4974-bfa4-c250e527314c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:18.872352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d4fd72-0a05-4974-bfa4-c250e527314c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:18.874206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d4fd72-0a05-4974-bfa4-c250e527314c service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:18.886517 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1148/2323] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:41:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:41:18.958193 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:18.959668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:19.031949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e0cd77-e034-4307-89f4-62dc25545d08 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:19.036631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:19.037251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:19.037532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:19.037803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb admin admin] PATH_INFO: `/v3/projects/e71c7904570b4e2689ac789184673f0d` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:19.039441 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb admin admin] RBAC: Authorizing `identity:delete_project(project_id=e71c7904570b4e2689ac789184673f0d)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:19.041378 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:19.206095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:41:19.209220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e71c7904570b4e2689ac789184673f0d', 'request_id': 'req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:19.216497 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e71c7904570b4e2689ac789184673f0d', 'request_id': 'req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:19.223576 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e71c7904570b4e2689ac789184673f0d', 'request_id': 'req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:19.234851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb admin admin] The token cache is being invalidate because project e71c7904570b4e2689ac789184673f0d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:41:19.235135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:19.235747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fb0cd2d8-e989-4ec2-9541-36c0ba026bfb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:41:19.237264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1176/2324] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:41:18 2022] DELETE /identity/v3/projects/e71c7904570b4e2689ac789184673f0d => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:41:19.242040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690f3b5f-960f-41e6-b575-dcf0d3b2aada None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:19.242529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690f3b5f-960f-41e6-b575-dcf0d3b2aada None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:19.242641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690f3b5f-960f-41e6-b575-dcf0d3b2aada None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:19.262139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-690f3b5f-960f-41e6-b575-dcf0d3b2aada None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:41:19.263325 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-690f3b5f-960f-41e6-b575-dcf0d3b2aada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:19.320715 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1149/2325] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:41:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:41:19.328453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88817c1b-feca-4c46-9af1-f57c42c3918b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:19.328733 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88817c1b-feca-4c46-9af1-f57c42c3918b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:19.329019 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88817c1b-feca-4c46-9af1-f57c42c3918b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:19.348330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-88817c1b-feca-4c46-9af1-f57c42c3918b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:41:19.349518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-88817c1b-feca-4c46-9af1-f57c42c3918b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:19.357351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1177/2326] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:41:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:41:19.361452 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690f3b5f-960f-41e6-b575-dcf0d3b2aada None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:19.370744 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690f3b5f-960f-41e6-b575-dcf0d3b2aada None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:19.375339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e47bc0-6072-4cad-9fef-161686574943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:19.375917 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e47bc0-6072-4cad-9fef-161686574943 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:19.376204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e47bc0-6072-4cad-9fef-161686574943 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:19.376500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e47bc0-6072-4cad-9fef-161686574943 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:19.377135 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e47bc0-6072-4cad-9fef-161686574943 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:41:19.377433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e47bc0-6072-4cad-9fef-161686574943 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:19.379039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e47bc0-6072-4cad-9fef-161686574943 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:19.390742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1150/2327] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:41:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:41:19.394814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88817c1b-feca-4c46-9af1-f57c42c3918b None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:19.404259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88817c1b-feca-4c46-9af1-f57c42c3918b None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:19.408777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54db3a54-09d2-4ae0-9631-9cfb90c496c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:19.409357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54db3a54-09d2-4ae0-9631-9cfb90c496c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:19.409630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54db3a54-09d2-4ae0-9631-9cfb90c496c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:19.409903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54db3a54-09d2-4ae0-9631-9cfb90c496c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:19.410525 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54db3a54-09d2-4ae0-9631-9cfb90c496c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:19.412412 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54db3a54-09d2-4ae0-9631-9cfb90c496c4 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:19.438177 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1178/2328] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:41:19 2022] POST /identity/v3/projects => generated 363 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:41:19.442110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e47bc0-6072-4cad-9fef-161686574943 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:19.452147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e47bc0-6072-4cad-9fef-161686574943 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:19.508097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fccfba4-e492-4bd2-9107-4d1f1c5d7d4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:19.508732 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fccfba4-e492-4bd2-9107-4d1f1c5d7d4a admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:19.509014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fccfba4-e492-4bd2-9107-4d1f1c5d7d4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:19.509290 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fccfba4-e492-4bd2-9107-4d1f1c5d7d4a admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:19.509914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fccfba4-e492-4bd2-9107-4d1f1c5d7d4a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:19.511832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fccfba4-e492-4bd2-9107-4d1f1c5d7d4a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:19.533409 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1151/2329] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:41:19 2022] POST /identity/v3/users => generated 282 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:41:19.537614 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54db3a54-09d2-4ae0-9631-9cfb90c496c4 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:19.547684 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54db3a54-09d2-4ae0-9631-9cfb90c496c4 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:19.552179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0a4346-7b0c-46ab-8a63-9fbbc86cc470 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:19.552785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0a4346-7b0c-46ab-8a63-9fbbc86cc470 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:19.553060 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0a4346-7b0c-46ab-8a63-9fbbc86cc470 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:19.553328 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0a4346-7b0c-46ab-8a63-9fbbc86cc470 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:19.553961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0a4346-7b0c-46ab-8a63-9fbbc86cc470 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:19.555513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0a4346-7b0c-46ab-8a63-9fbbc86cc470 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:19.565309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1179/2330] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:41:19 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:41:19.568989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fccfba4-e492-4bd2-9107-4d1f1c5d7d4a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:19.579033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fccfba4-e492-4bd2-9107-4d1f1c5d7d4a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:19.583697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337cb423-dde7-4120-9dca-33986e5a5b62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:19.584274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337cb423-dde7-4120-9dca-33986e5a5b62 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:19.584576 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337cb423-dde7-4120-9dca-33986e5a5b62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:19.584843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337cb423-dde7-4120-9dca-33986e5a5b62 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:19.586078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337cb423-dde7-4120-9dca-33986e5a5b62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:19.587651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337cb423-dde7-4120-9dca-33986e5a5b62 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:19.598304 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1152/2331] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:41:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:41:19.602359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0a4346-7b0c-46ab-8a63-9fbbc86cc470 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:19.612153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0a4346-7b0c-46ab-8a63-9fbbc86cc470 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:19.617123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5feac80-6728-49c9-b5d7-86c5fc4ad14b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:19.617753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5feac80-6728-49c9-b5d7-86c5fc4ad14b admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:19.618026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5feac80-6728-49c9-b5d7-86c5fc4ad14b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:19.618291 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5feac80-6728-49c9-b5d7-86c5fc4ad14b admin admin] PATH_INFO: `/v3/projects/f14e9c53ffff48eeb2918aaedd697832/users/63239eb27ecf4152b868e57575eb3fb8/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:19.633103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5feac80-6728-49c9-b5d7-86c5fc4ad14b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f14e9c53ffff48eeb2918aaedd697832, user_id=63239eb27ecf4152b868e57575eb3fb8, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:19.635712 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5feac80-6728-49c9-b5d7-86c5fc4ad14b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:19.649124 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1180/2332] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:41:19 2022] PUT /identity/v3/projects/f14e9c53ffff48eeb2918aaedd697832/users/63239eb27ecf4152b868e57575eb3fb8/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:41:19.659190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0df482-f653-4c6b-b805-64f763a5d71e None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:19.659471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0df482-f653-4c6b-b805-64f763a5d71e None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:19.659815 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0df482-f653-4c6b-b805-64f763a5d71e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:19.672747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:41:19.688331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-ce0df482-f653-4c6b-b805-64f763a5d71e None None] MFA Rules not processed for user `63239eb27ecf4152b868e57575eb3fb8`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:41:19.689546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-ce0df482-f653-4c6b-b805-64f763a5d71e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:19.751797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1153/2333] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:41:19 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:41:19.756128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5feac80-6728-49c9-b5d7-86c5fc4ad14b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:19.815496 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5feac80-6728-49c9-b5d7-86c5fc4ad14b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:19.820664 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa13fcfa-717e-4810-ac12-9dfed5d45d07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:19.821272 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa13fcfa-717e-4810-ac12-9dfed5d45d07 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:19.821547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa13fcfa-717e-4810-ac12-9dfed5d45d07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:19.821814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa13fcfa-717e-4810-ac12-9dfed5d45d07 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:19.822441 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa13fcfa-717e-4810-ac12-9dfed5d45d07 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:19.824300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa13fcfa-717e-4810-ac12-9dfed5d45d07 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:19.849714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1181/2334] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:41:19 2022] POST /identity/v3/projects => generated 365 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:41:19.853983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0df482-f653-4c6b-b805-64f763a5d71e None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:19.864914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0df482-f653-4c6b-b805-64f763a5d71e None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:19.920507 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f0be8f-e3bf-4d04-a8e4-ce89771b6f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:19.921100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f0be8f-e3bf-4d04-a8e4-ce89771b6f92 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:19.921378 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f0be8f-e3bf-4d04-a8e4-ce89771b6f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:19.921649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f0be8f-e3bf-4d04-a8e4-ce89771b6f92 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:19.922391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f0be8f-e3bf-4d04-a8e4-ce89771b6f92 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:19.924363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f0be8f-e3bf-4d04-a8e4-ce89771b6f92 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:19.946802 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1154/2335] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:41:19 2022] POST /identity/v3/users => generated 289 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:41:19.950912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa13fcfa-717e-4810-ac12-9dfed5d45d07 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:19.960877 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa13fcfa-717e-4810-ac12-9dfed5d45d07 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:19.965409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c784e5-5aa1-4750-955d-9ecc2e2409bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:19.965997 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c784e5-5aa1-4750-955d-9ecc2e2409bb admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:19.966272 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c784e5-5aa1-4750-955d-9ecc2e2409bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:19.966542 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c784e5-5aa1-4750-955d-9ecc2e2409bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:19.967181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c784e5-5aa1-4750-955d-9ecc2e2409bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:19.968795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c784e5-5aa1-4750-955d-9ecc2e2409bb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:19.978924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1182/2336] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:41:19 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:41:19.983116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f0be8f-e3bf-4d04-a8e4-ce89771b6f92 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:19.994150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f0be8f-e3bf-4d04-a8e4-ce89771b6f92 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:19.998843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ca2818-6e6c-4450-8568-8442df2a1ba7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:19.999479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ca2818-6e6c-4450-8568-8442df2a1ba7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:19.999759 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ca2818-6e6c-4450-8568-8442df2a1ba7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:20.000118 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ca2818-6e6c-4450-8568-8442df2a1ba7 admin admin] PATH_INFO: `/v3/projects/5cf3653f3389495ab0a2a0aaec4e7838/users/7412134423514766ac49d72b9d8df793/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:20.018854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ca2818-6e6c-4450-8568-8442df2a1ba7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5cf3653f3389495ab0a2a0aaec4e7838, user_id=7412134423514766ac49d72b9d8df793, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:20.022912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ca2818-6e6c-4450-8568-8442df2a1ba7 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:20.036916 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1155/2337] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:41:19 2022] PUT /identity/v3/projects/5cf3653f3389495ab0a2a0aaec4e7838/users/7412134423514766ac49d72b9d8df793/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:41:20.041318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c784e5-5aa1-4750-955d-9ecc2e2409bb admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:20.051095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c784e5-5aa1-4750-955d-9ecc2e2409bb admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:20.091547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:41:20.107826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44e162e-5efc-42cf-9e35-2bafaa95f83f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:20.108460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44e162e-5efc-42cf-9e35-2bafaa95f83f admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:20.108742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44e162e-5efc-42cf-9e35-2bafaa95f83f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:20.109013 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44e162e-5efc-42cf-9e35-2bafaa95f83f admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:20.109649 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44e162e-5efc-42cf-9e35-2bafaa95f83f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:20.111220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44e162e-5efc-42cf-9e35-2bafaa95f83f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:20.121247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1183/2338] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:41:20 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:41:20.125449 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ca2818-6e6c-4450-8568-8442df2a1ba7 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:41:20.135472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ca2818-6e6c-4450-8568-8442df2a1ba7 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:41:20.139936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3eb110-9995-427d-9db3-5c454b80f27d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:41:20.140595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3eb110-9995-427d-9db3-5c454b80f27d admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:20.140873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3eb110-9995-427d-9db3-5c454b80f27d admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:20.141141 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3eb110-9995-427d-9db3-5c454b80f27d admin admin] PATH_INFO: `/v3/projects/5cf3653f3389495ab0a2a0aaec4e7838/users/7412134423514766ac49d72b9d8df793/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:20.143021 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3eb110-9995-427d-9db3-5c454b80f27d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5cf3653f3389495ab0a2a0aaec4e7838, user_id=7412134423514766ac49d72b9d8df793, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:41:20.145664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3eb110-9995-427d-9db3-5c454b80f27d admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:41:20.155501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1156/2339] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:41:20 2022] PUT /identity/v3/projects/5cf3653f3389495ab0a2a0aaec4e7838/users/7412134423514766ac49d72b9d8df793/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 30 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:41:20.220446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd5c635-46e9-41bd-95a7-646f6cea5d07 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:41:20.220723 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd5c635-46e9-41bd-95a7-646f6cea5d07 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:41:20.220990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd5c635-46e9-41bd-95a7-646f6cea5d07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:41:20.250060 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-3cd5c635-46e9-41bd-95a7-646f6cea5d07 None None] MFA Rules not processed for user `7412134423514766ac49d72b9d8df793`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:41:20.251245 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-3cd5c635-46e9-41bd-95a7-646f6cea5d07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:41:20.315965 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1184/2340] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:41:20 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:41:30.151907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=73,keystone:INSERT=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:41:30.313115 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=43 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:42:20.386377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3eb110-9995-427d-9db3-5c454b80f27d admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:20.447263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3eb110-9995-427d-9db3-5c454b80f27d admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:20.452200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85b80cc-29b0-4671-937c-7d8160483b36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:20.452903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85b80cc-29b0-4671-937c-7d8160483b36 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:20.453183 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85b80cc-29b0-4671-937c-7d8160483b36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:20.453455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85b80cc-29b0-4671-937c-7d8160483b36 admin admin] PATH_INFO: `/v3/users/63239eb27ecf4152b868e57575eb3fb8` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:20.454571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85b80cc-29b0-4671-937c-7d8160483b36 admin admin] RBAC: Authorizing `identity:delete_user(user_id=63239eb27ecf4152b868e57575eb3fb8)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:20.456521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85b80cc-29b0-4671-937c-7d8160483b36 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:20.504687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c85b80cc-29b0-4671-937c-7d8160483b36 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '63239eb27ecf4152b868e57575eb3fb8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:20.505334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c85b80cc-29b0-4671-937c-7d8160483b36 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '63239eb27ecf4152b868e57575eb3fb8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:20.505927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c85b80cc-29b0-4671-937c-7d8160483b36 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '63239eb27ecf4152b868e57575eb3fb8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:20.513254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c85b80cc-29b0-4671-937c-7d8160483b36 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '63239eb27ecf4152b868e57575eb3fb8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:20.524540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c85b80cc-29b0-4671-937c-7d8160483b36 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '63239eb27ecf4152b868e57575eb3fb8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:20.536822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1157/2341] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:42:20 2022] DELETE /identity/v3/users/63239eb27ecf4152b868e57575eb3fb8 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:42:20.541302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ab6c8f-1f37-4cd0-9666-8ea07b1dca1d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:20.541795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ab6c8f-1f37-4cd0-9666-8ea07b1dca1d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:20.541909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ab6c8f-1f37-4cd0-9666-8ea07b1dca1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:20.560490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-a6ab6c8f-1f37-4cd0-9666-8ea07b1dca1d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:42:20.561669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a6ab6c8f-1f37-4cd0-9666-8ea07b1dca1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:42:20.618934 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1185/2342] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:42:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:42:20.629166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85b80cc-29b0-4671-937c-7d8160483b36 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:20.630547 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c85b80cc-29b0-4671-937c-7d8160483b36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:42:20.685281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85b80cc-29b0-4671-937c-7d8160483b36 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:20.688820 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc95b447-ed39-4f30-9e07-41da46887389 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:20.689408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc95b447-ed39-4f30-9e07-41da46887389 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:20.689681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc95b447-ed39-4f30-9e07-41da46887389 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:20.689952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc95b447-ed39-4f30-9e07-41da46887389 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:20.699168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc95b447-ed39-4f30-9e07-41da46887389 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:20.700619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc95b447-ed39-4f30-9e07-41da46887389 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:20.709641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1158/2343] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:42:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:42:20.973869 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ab6c8f-1f37-4cd0-9666-8ea07b1dca1d None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:20.975434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a6ab6c8f-1f37-4cd0-9666-8ea07b1dca1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:42:20.988023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ab6c8f-1f37-4cd0-9666-8ea07b1dca1d None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:20.992700 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:20.993859 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:20.994166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:20.994435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] PATH_INFO: `/v3/projects/f14e9c53ffff48eeb2918aaedd697832` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:20.995581 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f14e9c53ffff48eeb2918aaedd697832)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:20.997507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:21.052749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f14e9c53ffff48eeb2918aaedd697832', 'request_id': 'req-ce62f9d9-d230-4271-8109-e08f860deac5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.058730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f14e9c53ffff48eeb2918aaedd697832', 'request_id': 'req-ce62f9d9-d230-4271-8109-e08f860deac5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.066057 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f14e9c53ffff48eeb2918aaedd697832', 'request_id': 'req-ce62f9d9-d230-4271-8109-e08f860deac5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.077507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] The token cache is being invalidate because project f14e9c53ffff48eeb2918aaedd697832 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:42:21.077791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.078398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.079859 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1186/2344] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:42:20 2022] DELETE /identity/v3/projects/f14e9c53ffff48eeb2918aaedd697832 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:42:21.083810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc95b447-ed39-4f30-9e07-41da46887389 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:21.093253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc95b447-ed39-4f30-9e07-41da46887389 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:21.110870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:42:21.150482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a558a3c6-7fc8-47cb-85d2-f3c8c126c336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:21.151096 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a558a3c6-7fc8-47cb-85d2-f3c8c126c336 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:21.151370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a558a3c6-7fc8-47cb-85d2-f3c8c126c336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:21.151637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a558a3c6-7fc8-47cb-85d2-f3c8c126c336 admin admin] PATH_INFO: `/v3/users/7412134423514766ac49d72b9d8df793` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:21.152771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a558a3c6-7fc8-47cb-85d2-f3c8c126c336 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7412134423514766ac49d72b9d8df793)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:21.154688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a558a3c6-7fc8-47cb-85d2-f3c8c126c336 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:21.205336 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a558a3c6-7fc8-47cb-85d2-f3c8c126c336 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7412134423514766ac49d72b9d8df793'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.206044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a558a3c6-7fc8-47cb-85d2-f3c8c126c336 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7412134423514766ac49d72b9d8df793'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.207290 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a558a3c6-7fc8-47cb-85d2-f3c8c126c336 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7412134423514766ac49d72b9d8df793'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.214680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a558a3c6-7fc8-47cb-85d2-f3c8c126c336 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7412134423514766ac49d72b9d8df793'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.226463 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a558a3c6-7fc8-47cb-85d2-f3c8c126c336 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7412134423514766ac49d72b9d8df793'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.239491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1159/2345] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:42:21 2022] DELETE /identity/v3/users/7412134423514766ac49d72b9d8df793 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:42:21.504890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:21.506307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:42:21.519322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce62f9d9-d230-4271-8109-e08f860deac5 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:21.575657 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3e5640-5462-46f3-84a3-42027aefeec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:21.576323 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3e5640-5462-46f3-84a3-42027aefeec4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:21.576607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3e5640-5462-46f3-84a3-42027aefeec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:21.576873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3e5640-5462-46f3-84a3-42027aefeec4 admin admin] PATH_INFO: `/v3/projects/5cf3653f3389495ab0a2a0aaec4e7838` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:21.577995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3e5640-5462-46f3-84a3-42027aefeec4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5cf3653f3389495ab0a2a0aaec4e7838)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:21.579913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3e5640-5462-46f3-84a3-42027aefeec4 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:21.609237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:42:21.626977 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-2f3e5640-5462-46f3-84a3-42027aefeec4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5cf3653f3389495ab0a2a0aaec4e7838', 'request_id': 'req-2f3e5640-5462-46f3-84a3-42027aefeec4'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.632406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-2f3e5640-5462-46f3-84a3-42027aefeec4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5cf3653f3389495ab0a2a0aaec4e7838', 'request_id': 'req-2f3e5640-5462-46f3-84a3-42027aefeec4'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.639369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-2f3e5640-5462-46f3-84a3-42027aefeec4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5cf3653f3389495ab0a2a0aaec4e7838', 'request_id': 'req-2f3e5640-5462-46f3-84a3-42027aefeec4'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.650211 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-2f3e5640-5462-46f3-84a3-42027aefeec4 admin admin] The token cache is being invalidate because project 5cf3653f3389495ab0a2a0aaec4e7838 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:42:21.650504 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-2f3e5640-5462-46f3-84a3-42027aefeec4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.651133 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-2f3e5640-5462-46f3-84a3-42027aefeec4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:42:21.652631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1187/2346] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:42:21 2022] DELETE /identity/v3/projects/5cf3653f3389495ab0a2a0aaec4e7838 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:42:21.658241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950e93b1-5e20-43ed-b82e-db0f38292133 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:21.658701 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950e93b1-5e20-43ed-b82e-db0f38292133 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:21.658825 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950e93b1-5e20-43ed-b82e-db0f38292133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:21.678734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-950e93b1-5e20-43ed-b82e-db0f38292133 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:42:21.679919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-950e93b1-5e20-43ed-b82e-db0f38292133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:42:21.737514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1160/2347] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:42:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:42:21.745794 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c8e170-290d-420f-9ea5-cf3f5c898fbe None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:21.746080 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c8e170-290d-420f-9ea5-cf3f5c898fbe None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:21.746383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c8e170-290d-420f-9ea5-cf3f5c898fbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:21.765551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-27c8e170-290d-420f-9ea5-cf3f5c898fbe None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:42:21.766747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-27c8e170-290d-420f-9ea5-cf3f5c898fbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:42:21.774471 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1188/2348] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:42:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:42:21.779040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950e93b1-5e20-43ed-b82e-db0f38292133 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:21.788662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950e93b1-5e20-43ed-b82e-db0f38292133 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:21.794056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bad374-dea3-4776-874c-460464d2a6c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:21.794682 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bad374-dea3-4776-874c-460464d2a6c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:21.794961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bad374-dea3-4776-874c-460464d2a6c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:21.795232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bad374-dea3-4776-874c-460464d2a6c7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:21.795885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5bad374-dea3-4776-874c-460464d2a6c7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:42:21.796189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5bad374-dea3-4776-874c-460464d2a6c7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:21.797832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5bad374-dea3-4776-874c-460464d2a6c7 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:21.809487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1161/2349] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:42:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:42:21.813612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c8e170-290d-420f-9ea5-cf3f5c898fbe None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:21.822709 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c8e170-290d-420f-9ea5-cf3f5c898fbe None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:21.827363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7ceef8-dd1e-4e3b-a9ce-91bba0342e2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:21.827949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7ceef8-dd1e-4e3b-a9ce-91bba0342e2f admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:21.828228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7ceef8-dd1e-4e3b-a9ce-91bba0342e2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:21.828529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7ceef8-dd1e-4e3b-a9ce-91bba0342e2f admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:21.829166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7ceef8-dd1e-4e3b-a9ce-91bba0342e2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:21.831041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7ceef8-dd1e-4e3b-a9ce-91bba0342e2f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:21.859392 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1189/2350] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:42:21 2022] POST /identity/v3/projects => generated 387 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:42:21.863313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bad374-dea3-4776-874c-460464d2a6c7 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:21.873206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bad374-dea3-4776-874c-460464d2a6c7 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:21.897981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:42:21.930154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5567fbcc-b444-4858-955d-1d36d7263117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:21.930754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5567fbcc-b444-4858-955d-1d36d7263117 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:21.931030 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5567fbcc-b444-4858-955d-1d36d7263117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:21.931328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5567fbcc-b444-4858-955d-1d36d7263117 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:21.931985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5567fbcc-b444-4858-955d-1d36d7263117 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:21.934081 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5567fbcc-b444-4858-955d-1d36d7263117 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:21.954287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1162/2351] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:42:21 2022] POST /identity/v3/users => generated 294 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:42:21.958438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7ceef8-dd1e-4e3b-a9ce-91bba0342e2f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:21.969394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7ceef8-dd1e-4e3b-a9ce-91bba0342e2f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:21.974294 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036250ef-3be3-4764-b451-244f36402298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:21.974882 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036250ef-3be3-4764-b451-244f36402298 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:21.975157 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036250ef-3be3-4764-b451-244f36402298 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:21.975421 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036250ef-3be3-4764-b451-244f36402298 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:21.976058 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036250ef-3be3-4764-b451-244f36402298 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:21.978240 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036250ef-3be3-4764-b451-244f36402298 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:21.988373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1190/2352] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:42:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:42:21.992098 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5567fbcc-b444-4858-955d-1d36d7263117 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:22.001683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5567fbcc-b444-4858-955d-1d36d7263117 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:22.006219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c69618-77b6-4ab7-ab4f-706137858f11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:22.006796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c69618-77b6-4ab7-ab4f-706137858f11 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:22.007074 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c69618-77b6-4ab7-ab4f-706137858f11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:22.007350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c69618-77b6-4ab7-ab4f-706137858f11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:22.007980 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c69618-77b6-4ab7-ab4f-706137858f11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:22.009608 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c69618-77b6-4ab7-ab4f-706137858f11 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:22.019574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1163/2353] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:42:21 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:42:22.023524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036250ef-3be3-4764-b451-244f36402298 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:22.033785 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036250ef-3be3-4764-b451-244f36402298 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:22.038281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26fe91b-b7b9-421e-a668-78be673c6c51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:22.038913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26fe91b-b7b9-421e-a668-78be673c6c51 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:22.039190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26fe91b-b7b9-421e-a668-78be673c6c51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:22.039458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26fe91b-b7b9-421e-a668-78be673c6c51 admin admin] PATH_INFO: `/v3/projects/2587c7d3fd8549a0bb79444d30d8f38f/users/e8786dcc5afc451481506603efec8641/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:22.054191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26fe91b-b7b9-421e-a668-78be673c6c51 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2587c7d3fd8549a0bb79444d30d8f38f, user_id=e8786dcc5afc451481506603efec8641, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:22.056931 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26fe91b-b7b9-421e-a668-78be673c6c51 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:22.069062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1191/2354] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:42:22 2022] PUT /identity/v3/projects/2587c7d3fd8549a0bb79444d30d8f38f/users/e8786dcc5afc451481506603efec8641/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:42:22.078582 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66566137-a9f2-4da1-8f15-7d178242d999 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:22.078873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66566137-a9f2-4da1-8f15-7d178242d999 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:22.079154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66566137-a9f2-4da1-8f15-7d178242d999 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:22.107360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-66566137-a9f2-4da1-8f15-7d178242d999 None None] MFA Rules not processed for user `e8786dcc5afc451481506603efec8641`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:42:22.108594 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-66566137-a9f2-4da1-8f15-7d178242d999 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:42:22.170534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1164/2355] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:42:22 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:42:22.174975 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26fe91b-b7b9-421e-a668-78be673c6c51 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:22.235097 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26fe91b-b7b9-421e-a668-78be673c6c51 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:22.239763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25e94ab-b181-4080-a541-90abef0601de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:22.240382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25e94ab-b181-4080-a541-90abef0601de admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:22.240665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25e94ab-b181-4080-a541-90abef0601de admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:22.240939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25e94ab-b181-4080-a541-90abef0601de admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:22.241568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25e94ab-b181-4080-a541-90abef0601de admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:22.243483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25e94ab-b181-4080-a541-90abef0601de admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:22.269326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1192/2356] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:42:22 2022] POST /identity/v3/projects => generated 387 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:42:22.273302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66566137-a9f2-4da1-8f15-7d178242d999 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:22.284372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66566137-a9f2-4da1-8f15-7d178242d999 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:22.338276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d62bd2-c5c6-48bf-ac14-5d7365e92df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:22.339045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d62bd2-c5c6-48bf-ac14-5d7365e92df7 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:22.339416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d62bd2-c5c6-48bf-ac14-5d7365e92df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:22.339696 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d62bd2-c5c6-48bf-ac14-5d7365e92df7 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:22.340541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d62bd2-c5c6-48bf-ac14-5d7365e92df7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:22.342523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d62bd2-c5c6-48bf-ac14-5d7365e92df7 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:22.362944 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1165/2357] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:42:22 2022] POST /identity/v3/users => generated 300 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:42:22.366858 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25e94ab-b181-4080-a541-90abef0601de admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:22.377067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25e94ab-b181-4080-a541-90abef0601de admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:22.382129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7488a64f-4e30-4981-ae05-a6fe93769a6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:22.382717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7488a64f-4e30-4981-ae05-a6fe93769a6b admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:22.382995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7488a64f-4e30-4981-ae05-a6fe93769a6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:22.383267 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7488a64f-4e30-4981-ae05-a6fe93769a6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:22.384415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7488a64f-4e30-4981-ae05-a6fe93769a6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:22.385989 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7488a64f-4e30-4981-ae05-a6fe93769a6b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:22.396228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1193/2358] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:42:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:42:22.400797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d62bd2-c5c6-48bf-ac14-5d7365e92df7 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:22.410733 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d62bd2-c5c6-48bf-ac14-5d7365e92df7 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:22.415291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a44a4d-4efe-4b4d-a4d4-70cfc144a1af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:22.415924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a44a4d-4efe-4b4d-a4d4-70cfc144a1af admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:22.416200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a44a4d-4efe-4b4d-a4d4-70cfc144a1af admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:22.416502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a44a4d-4efe-4b4d-a4d4-70cfc144a1af admin admin] PATH_INFO: `/v3/projects/23f8a735152f4dffa51b30dc1de2579f/users/1201a5092555404693f2303a4b50f18e/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:22.431714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a44a4d-4efe-4b4d-a4d4-70cfc144a1af admin admin] RBAC: Authorizing `identity:create_grant(project_id=23f8a735152f4dffa51b30dc1de2579f, user_id=1201a5092555404693f2303a4b50f18e, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:22.434532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a44a4d-4efe-4b4d-a4d4-70cfc144a1af admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:22.447113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1166/2359] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:42:22 2022] PUT /identity/v3/projects/23f8a735152f4dffa51b30dc1de2579f/users/1201a5092555404693f2303a4b50f18e/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:42:22.450913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7488a64f-4e30-4981-ae05-a6fe93769a6b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:22.460502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7488a64f-4e30-4981-ae05-a6fe93769a6b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:22.480033 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:42:22.521955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4d7714-936b-4c2e-b9eb-a68d2565a13a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:22.522572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4d7714-936b-4c2e-b9eb-a68d2565a13a admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:22.522950 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4d7714-936b-4c2e-b9eb-a68d2565a13a admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:22.523222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4d7714-936b-4c2e-b9eb-a68d2565a13a admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:22.523855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4d7714-936b-4c2e-b9eb-a68d2565a13a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:22.525446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4d7714-936b-4c2e-b9eb-a68d2565a13a admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:22.535006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1194/2360] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:42:22 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:42:22.540608 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a44a4d-4efe-4b4d-a4d4-70cfc144a1af admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:42:22.546212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:42:22.552324 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a44a4d-4efe-4b4d-a4d4-70cfc144a1af admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:42:22.558105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d76899f-fb30-4e20-b9b5-2f32b1ca80cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:42:22.558743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d76899f-fb30-4e20-b9b5-2f32b1ca80cd admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:22.559019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d76899f-fb30-4e20-b9b5-2f32b1ca80cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:22.559286 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d76899f-fb30-4e20-b9b5-2f32b1ca80cd admin admin] PATH_INFO: `/v3/projects/23f8a735152f4dffa51b30dc1de2579f/users/1201a5092555404693f2303a4b50f18e/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:22.561198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d76899f-fb30-4e20-b9b5-2f32b1ca80cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=23f8a735152f4dffa51b30dc1de2579f, user_id=1201a5092555404693f2303a4b50f18e, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:42:22.563835 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d76899f-fb30-4e20-b9b5-2f32b1ca80cd admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:42:22.574412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1167/2361] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:42:22 2022] PUT /identity/v3/projects/23f8a735152f4dffa51b30dc1de2579f/users/1201a5092555404693f2303a4b50f18e/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:42:22.584170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab517b5-03b1-4a80-9c3e-68f34a1602c5 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:42:22.584649 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab517b5-03b1-4a80-9c3e-68f34a1602c5 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:42:22.584764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab517b5-03b1-4a80-9c3e-68f34a1602c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:42:22.612020 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-fab517b5-03b1-4a80-9c3e-68f34a1602c5 None None] MFA Rules not processed for user `1201a5092555404693f2303a4b50f18e`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:42:22.613262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-fab517b5-03b1-4a80-9c3e-68f34a1602c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:42:22.679435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1195/2362] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:42:22 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:42:32.570407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=4,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:42:32.676643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=52 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:43:17.460910 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d76899f-fb30-4e20-b9b5-2f32b1ca80cd admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:17.462800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-4d76899f-fb30-4e20-b9b5-2f32b1ca80cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:17.524467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d76899f-fb30-4e20-b9b5-2f32b1ca80cd admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:17.528054 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cbd2dd-f648-47fe-b89b-f1070bf1d4fe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:17.528687 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cbd2dd-f648-47fe-b89b-f1070bf1d4fe service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:17.528964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cbd2dd-f648-47fe-b89b-f1070bf1d4fe service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:17.529233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cbd2dd-f648-47fe-b89b-f1070bf1d4fe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:17.530959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-25cbd2dd-f648-47fe-b89b-f1070bf1d4fe service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:17.612843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cbd2dd-f648-47fe-b89b-f1070bf1d4fe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:17.614294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cbd2dd-f648-47fe-b89b-f1070bf1d4fe service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:17.632818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1168/2363] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:43:17 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:43:17.727341 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab517b5-03b1-4a80-9c3e-68f34a1602c5 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:17.737022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab517b5-03b1-4a80-9c3e-68f34a1602c5 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:17.740545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe4eb39-13fd-41a0-85e1-2933cbd53d72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:17.741131 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe4eb39-13fd-41a0-85e1-2933cbd53d72 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:17.741404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe4eb39-13fd-41a0-85e1-2933cbd53d72 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:17.741669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe4eb39-13fd-41a0-85e1-2933cbd53d72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:17.743996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-9fe4eb39-13fd-41a0-85e1-2933cbd53d72 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-9fe4eb39-13fd-41a0-85e1-2933cbd53d72 service cinder] Could not find project: 5cf3653f3389495ab0a2a0aaec4e7838.: keystone.exception.ProjectNotFound: Could not find project: 5cf3653f3389495ab0a2a0aaec4e7838. Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/auth.py", line 285, in get Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') Sep 28 12:43:17.771726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 145, in validate_token Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application token = self._validate_token(token_id) Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:43:17.772748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 179, in _validate_token Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application token.mint(token_id, issued_at) Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 579, in mint Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._validate_token_resources() Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 464, in _validate_token_resources Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application if self.project and not self.project.get('enabled'): Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 162, in project Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self.__project = PROVIDERS.resource_api.get_project( Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:43:17.774070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:43:17.775105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:43:17.776105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: 5cf3653f3389495ab0a2a0aaec4e7838. Sep 28 12:43:17.776105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:43:17.776105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1196/2364] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:43:17 2022] GET /identity/v3/auth/tokens => generated 113 bytes in 46 msecs (HTTP/1.1 404) 5 headers in 184 bytes (1 switches on core 0) Sep 28 12:43:17.955218 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cbd2dd-f648-47fe-b89b-f1070bf1d4fe service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:17.965722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cbd2dd-f648-47fe-b89b-f1070bf1d4fe service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:17.970642 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7c04fc-76eb-460c-9941-7feb2890b64d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:17.971276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7c04fc-76eb-460c-9941-7feb2890b64d service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:17.971579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7c04fc-76eb-460c-9941-7feb2890b64d service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:17.971979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7c04fc-76eb-460c-9941-7feb2890b64d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:17.982525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7c04fc-76eb-460c-9941-7feb2890b64d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:17.983944 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7c04fc-76eb-460c-9941-7feb2890b64d service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:17.994218 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1169/2365] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:43:17 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:43:18.022184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe4eb39-13fd-41a0-85e1-2933cbd53d72 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:18.095467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe4eb39-13fd-41a0-85e1-2933cbd53d72 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:18.100212 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba69097-3322-4a47-8aa8-3b6ec7eacefb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:18.100866 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba69097-3322-4a47-8aa8-3b6ec7eacefb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:18.101147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba69097-3322-4a47-8aa8-3b6ec7eacefb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:18.101989 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba69097-3322-4a47-8aa8-3b6ec7eacefb admin admin] PATH_INFO: `/v3/users/e8786dcc5afc451481506603efec8641` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:18.103129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba69097-3322-4a47-8aa8-3b6ec7eacefb admin admin] RBAC: Authorizing `identity:delete_user(user_id=e8786dcc5afc451481506603efec8641)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:18.105068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba69097-3322-4a47-8aa8-3b6ec7eacefb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:18.140492 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7c04fc-76eb-460c-9941-7feb2890b64d service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:18.141962 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-7c7c04fc-76eb-460c-9941-7feb2890b64d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:18.152424 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fba69097-3322-4a47-8aa8-3b6ec7eacefb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e8786dcc5afc451481506603efec8641'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.153093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fba69097-3322-4a47-8aa8-3b6ec7eacefb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e8786dcc5afc451481506603efec8641'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.159907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fba69097-3322-4a47-8aa8-3b6ec7eacefb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e8786dcc5afc451481506603efec8641'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.160932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fba69097-3322-4a47-8aa8-3b6ec7eacefb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e8786dcc5afc451481506603efec8641'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.173300 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-fba69097-3322-4a47-8aa8-3b6ec7eacefb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e8786dcc5afc451481506603efec8641'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.186392 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1197/2366] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:43:18 2022] DELETE /identity/v3/users/e8786dcc5afc451481506603efec8641 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:43:18.190811 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076ca1b6-7859-4bc1-9231-ddfc711974aa None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:18.191096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076ca1b6-7859-4bc1-9231-ddfc711974aa None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:18.191368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076ca1b6-7859-4bc1-9231-ddfc711974aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:18.202036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7c04fc-76eb-460c-9941-7feb2890b64d service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:18.209728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-076ca1b6-7859-4bc1-9231-ddfc711974aa None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:43:18.211446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-076ca1b6-7859-4bc1-9231-ddfc711974aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:18.227936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:43:18.251434 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e060ed3-ab88-43f5-a600-42b40640fae1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:18.252033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e060ed3-ab88-43f5-a600-42b40640fae1 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:18.252333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e060ed3-ab88-43f5-a600-42b40640fae1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:18.252613 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e060ed3-ab88-43f5-a600-42b40640fae1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:18.254439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5e060ed3-ab88-43f5-a600-42b40640fae1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:18.269377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1198/2367] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:43:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:43:18.279646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076ca1b6-7859-4bc1-9231-ddfc711974aa None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:18.281106 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-076ca1b6-7859-4bc1-9231-ddfc711974aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:18.293677 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076ca1b6-7859-4bc1-9231-ddfc711974aa None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:18.297320 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477a5219-7157-4ecf-aa3b-a2953423c076 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:18.297914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477a5219-7157-4ecf-aa3b-a2953423c076 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:18.298196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477a5219-7157-4ecf-aa3b-a2953423c076 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:18.298471 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477a5219-7157-4ecf-aa3b-a2953423c076 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:18.307589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477a5219-7157-4ecf-aa3b-a2953423c076 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:18.309046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477a5219-7157-4ecf-aa3b-a2953423c076 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:18.312911 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:43:18.320622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1199/2368] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:43:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:43:18.325872 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e060ed3-ab88-43f5-a600-42b40640fae1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:18.327282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e060ed3-ab88-43f5-a600-42b40640fae1 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:18.339414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1170/2369] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:43:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:43:18.670847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477a5219-7157-4ecf-aa3b-a2953423c076 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:18.675103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-477a5219-7157-4ecf-aa3b-a2953423c076 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:18.700384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477a5219-7157-4ecf-aa3b-a2953423c076 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:18.705440 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:18.706111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:18.706392 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:18.706662 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] PATH_INFO: `/v3/projects/2587c7d3fd8549a0bb79444d30d8f38f` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:18.707763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2587c7d3fd8549a0bb79444d30d8f38f)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:18.709683 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:18.761758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2587c7d3fd8549a0bb79444d30d8f38f', 'request_id': 'req-b4ee81e5-8453-495b-9e33-fac0f5f096b3'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.768550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2587c7d3fd8549a0bb79444d30d8f38f', 'request_id': 'req-b4ee81e5-8453-495b-9e33-fac0f5f096b3'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.777095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2587c7d3fd8549a0bb79444d30d8f38f', 'request_id': 'req-b4ee81e5-8453-495b-9e33-fac0f5f096b3'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.789487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] The token cache is being invalidate because project 2587c7d3fd8549a0bb79444d30d8f38f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:43:18.789726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.790433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.792067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1200/2370] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:43:18 2022] DELETE /identity/v3/projects/2587c7d3fd8549a0bb79444d30d8f38f => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:43:18.796232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e060ed3-ab88-43f5-a600-42b40640fae1 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:18.807041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e060ed3-ab88-43f5-a600-42b40640fae1 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:18.866522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f060440-8acb-41de-a6fe-3d31f87e03b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:18.867862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f060440-8acb-41de-a6fe-3d31f87e03b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:18.868349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f060440-8acb-41de-a6fe-3d31f87e03b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:18.870374 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f060440-8acb-41de-a6fe-3d31f87e03b8 admin admin] PATH_INFO: `/v3/users/1201a5092555404693f2303a4b50f18e` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:18.871677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f060440-8acb-41de-a6fe-3d31f87e03b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1201a5092555404693f2303a4b50f18e)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:18.873509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f060440-8acb-41de-a6fe-3d31f87e03b8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:18.937390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0f060440-8acb-41de-a6fe-3d31f87e03b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1201a5092555404693f2303a4b50f18e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.938059 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0f060440-8acb-41de-a6fe-3d31f87e03b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1201a5092555404693f2303a4b50f18e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.938674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0f060440-8acb-41de-a6fe-3d31f87e03b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1201a5092555404693f2303a4b50f18e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.946431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0f060440-8acb-41de-a6fe-3d31f87e03b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1201a5092555404693f2303a4b50f18e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.958619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:43:18.960093 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0f060440-8acb-41de-a6fe-3d31f87e03b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1201a5092555404693f2303a4b50f18e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:18.977658 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1171/2371] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:43:18 2022] DELETE /identity/v3/users/1201a5092555404693f2303a4b50f18e => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:43:19.248620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:19.250076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:19.276357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ee81e5-8453-495b-9e33-fac0f5f096b3 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:19.334803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a7f93b-dba1-46c4-bbec-e84adb693049 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:19.335524 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a7f93b-dba1-46c4-bbec-e84adb693049 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:19.335876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a7f93b-dba1-46c4-bbec-e84adb693049 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:19.336298 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a7f93b-dba1-46c4-bbec-e84adb693049 admin admin] PATH_INFO: `/v3/projects/23f8a735152f4dffa51b30dc1de2579f` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:19.337987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a7f93b-dba1-46c4-bbec-e84adb693049 admin admin] RBAC: Authorizing `identity:delete_project(project_id=23f8a735152f4dffa51b30dc1de2579f)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:19.339558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a7f93b-dba1-46c4-bbec-e84adb693049 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:19.390925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-77a7f93b-dba1-46c4-bbec-e84adb693049 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '23f8a735152f4dffa51b30dc1de2579f', 'request_id': 'req-77a7f93b-dba1-46c4-bbec-e84adb693049'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:19.396508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-77a7f93b-dba1-46c4-bbec-e84adb693049 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '23f8a735152f4dffa51b30dc1de2579f', 'request_id': 'req-77a7f93b-dba1-46c4-bbec-e84adb693049'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:19.407832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-77a7f93b-dba1-46c4-bbec-e84adb693049 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '23f8a735152f4dffa51b30dc1de2579f', 'request_id': 'req-77a7f93b-dba1-46c4-bbec-e84adb693049'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:19.421598 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-77a7f93b-dba1-46c4-bbec-e84adb693049 admin admin] The token cache is being invalidate because project 23f8a735152f4dffa51b30dc1de2579f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:43:19.421891 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-77a7f93b-dba1-46c4-bbec-e84adb693049 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:19.422575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-77a7f93b-dba1-46c4-bbec-e84adb693049 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:43:19.424125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1201/2372] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:43:19 2022] DELETE /identity/v3/projects/23f8a735152f4dffa51b30dc1de2579f => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:43:19.432213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0ae570-d786-433d-9d58-24269dcb1d98 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:19.433096 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0ae570-d786-433d-9d58-24269dcb1d98 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:19.433382 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0ae570-d786-433d-9d58-24269dcb1d98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:19.452648 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-ae0ae570-d786-433d-9d58-24269dcb1d98 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:43:19.453892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-ae0ae570-d786-433d-9d58-24269dcb1d98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:19.521095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1172/2373] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:43:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:43:19.530233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c76aaf-e239-477e-9711-899519217766 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:19.530513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c76aaf-e239-477e-9711-899519217766 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:19.530786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c76aaf-e239-477e-9711-899519217766 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:19.552242 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-15c76aaf-e239-477e-9711-899519217766 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:43:19.556265 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-15c76aaf-e239-477e-9711-899519217766 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:19.569957 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1202/2374] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:43:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:43:19.570515 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0ae570-d786-433d-9d58-24269dcb1d98 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:19.577647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c76aaf-e239-477e-9711-899519217766 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:19.581065 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-15c76aaf-e239-477e-9711-899519217766 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:19.581726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0ae570-d786-433d-9d58-24269dcb1d98 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:19.591633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:43:19.592808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22392e60-1cbb-4e55-8241-84662471b40d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:19.593831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22392e60-1cbb-4e55-8241-84662471b40d admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:19.594313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22392e60-1cbb-4e55-8241-84662471b40d admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:19.594775 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22392e60-1cbb-4e55-8241-84662471b40d admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:19.595610 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22392e60-1cbb-4e55-8241-84662471b40d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:43:19.596588 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22392e60-1cbb-4e55-8241-84662471b40d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:19.599625 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22392e60-1cbb-4e55-8241-84662471b40d admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:19.628185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1173/2375] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:43:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:43:19.635077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22392e60-1cbb-4e55-8241-84662471b40d admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:19.650532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22392e60-1cbb-4e55-8241-84662471b40d admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:19.676642 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c76aaf-e239-477e-9711-899519217766 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:19.679548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fa8cbd-693c-4fb9-b0f4-89b3bbfbc5fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:19.680781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fa8cbd-693c-4fb9-b0f4-89b3bbfbc5fd admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:19.681069 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fa8cbd-693c-4fb9-b0f4-89b3bbfbc5fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:19.685541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fa8cbd-693c-4fb9-b0f4-89b3bbfbc5fd admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:19.686848 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fa8cbd-693c-4fb9-b0f4-89b3bbfbc5fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:19.689736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fa8cbd-693c-4fb9-b0f4-89b3bbfbc5fd admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:19.694617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13f78d2-b1d0-40b1-addf-1d1da14004bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:19.695385 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f78d2-b1d0-40b1-addf-1d1da14004bf service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:19.696327 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f78d2-b1d0-40b1-addf-1d1da14004bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:19.696693 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f78d2-b1d0-40b1-addf-1d1da14004bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:19.698881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a13f78d2-b1d0-40b1-addf-1d1da14004bf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:19.749589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1174/2376] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:43:19 2022] POST /identity/v3/projects => generated 377 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:43:19.751193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fa8cbd-693c-4fb9-b0f4-89b3bbfbc5fd admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:19.802946 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13f78d2-b1d0-40b1-addf-1d1da14004bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:19.807793 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13f78d2-b1d0-40b1-addf-1d1da14004bf service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:19.822106 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1203/2377] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:43:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:43:19.856752 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fa8cbd-693c-4fb9-b0f4-89b3bbfbc5fd admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:19.877548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4a025c-aed6-49f9-a39c-37f4aa7ea727 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:19.878399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4a025c-aed6-49f9-a39c-37f4aa7ea727 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:19.878838 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4a025c-aed6-49f9-a39c-37f4aa7ea727 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:19.879216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4a025c-aed6-49f9-a39c-37f4aa7ea727 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:19.879961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4a025c-aed6-49f9-a39c-37f4aa7ea727 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:19.882406 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4a025c-aed6-49f9-a39c-37f4aa7ea727 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:19.919167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1175/2378] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:43:19 2022] POST /identity/v3/users => generated 289 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:43:19.924508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13f78d2-b1d0-40b1-addf-1d1da14004bf service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:19.935439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13f78d2-b1d0-40b1-addf-1d1da14004bf service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:19.940142 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76387fb-00d7-4edb-8579-f62439df356b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:19.940774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76387fb-00d7-4edb-8579-f62439df356b admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:19.941045 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76387fb-00d7-4edb-8579-f62439df356b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:19.941314 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76387fb-00d7-4edb-8579-f62439df356b admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:19.941957 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76387fb-00d7-4edb-8579-f62439df356b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:19.943509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76387fb-00d7-4edb-8579-f62439df356b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:19.956741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1204/2379] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:43:19 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:43:19.960020 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4a025c-aed6-49f9-a39c-37f4aa7ea727 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:19.971593 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4a025c-aed6-49f9-a39c-37f4aa7ea727 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:19.976559 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b803c3-2b56-4d19-a22d-03aa16b7ae2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:19.977160 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b803c3-2b56-4d19-a22d-03aa16b7ae2c admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:19.977435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b803c3-2b56-4d19-a22d-03aa16b7ae2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:19.977703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b803c3-2b56-4d19-a22d-03aa16b7ae2c admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:19.978345 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b803c3-2b56-4d19-a22d-03aa16b7ae2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:19.980038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b803c3-2b56-4d19-a22d-03aa16b7ae2c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:19.991195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1176/2380] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:43:19 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:43:19.995504 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76387fb-00d7-4edb-8579-f62439df356b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:20.005361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76387fb-00d7-4edb-8579-f62439df356b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:20.010554 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df86d00-643e-430c-8041-d0d474ab570b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:20.011204 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df86d00-643e-430c-8041-d0d474ab570b admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:20.011482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df86d00-643e-430c-8041-d0d474ab570b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:20.011753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df86d00-643e-430c-8041-d0d474ab570b admin admin] PATH_INFO: `/v3/projects/b0eb61ddcc874275a804bfc6df1e5829/users/0b5ecdca5a0543a1894e6462c3f2844f/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:20.030580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df86d00-643e-430c-8041-d0d474ab570b admin admin] RBAC: Authorizing `identity:create_grant(project_id=b0eb61ddcc874275a804bfc6df1e5829, user_id=0b5ecdca5a0543a1894e6462c3f2844f, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:20.033528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df86d00-643e-430c-8041-d0d474ab570b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:20.049290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1205/2381] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:43:19 2022] PUT /identity/v3/projects/b0eb61ddcc874275a804bfc6df1e5829/users/0b5ecdca5a0543a1894e6462c3f2844f/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:43:20.060837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c0d1bd-bbfc-45db-88f8-153a55bd4470 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:20.061310 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c0d1bd-bbfc-45db-88f8-153a55bd4470 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:20.061750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c0d1bd-bbfc-45db-88f8-153a55bd4470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:20.108467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-28c0d1bd-bbfc-45db-88f8-153a55bd4470 None None] MFA Rules not processed for user `0b5ecdca5a0543a1894e6462c3f2844f`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:43:20.109794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-28c0d1bd-bbfc-45db-88f8-153a55bd4470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:20.131424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:43:20.182031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1177/2382] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:43:20 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:43:20.186596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df86d00-643e-430c-8041-d0d474ab570b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:20.261597 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df86d00-643e-430c-8041-d0d474ab570b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:20.266556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9cb47c-8fdf-47cb-9a1b-90c3e6847a08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:20.267165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9cb47c-8fdf-47cb-9a1b-90c3e6847a08 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:20.267452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9cb47c-8fdf-47cb-9a1b-90c3e6847a08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:20.267709 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9cb47c-8fdf-47cb-9a1b-90c3e6847a08 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:20.268363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9cb47c-8fdf-47cb-9a1b-90c3e6847a08 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:20.270250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9cb47c-8fdf-47cb-9a1b-90c3e6847a08 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:20.286449 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:43:20.300026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1206/2383] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:43:20 2022] POST /identity/v3/projects => generated 379 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:43:20.303839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c0d1bd-bbfc-45db-88f8-153a55bd4470 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:20.314985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c0d1bd-bbfc-45db-88f8-153a55bd4470 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:20.371411 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4aa1bd-d558-43db-973d-b7d93cfea71a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:20.372015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4aa1bd-d558-43db-973d-b7d93cfea71a admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:20.372299 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4aa1bd-d558-43db-973d-b7d93cfea71a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:20.372605 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4aa1bd-d558-43db-973d-b7d93cfea71a admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:20.373232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4aa1bd-d558-43db-973d-b7d93cfea71a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:20.375148 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4aa1bd-d558-43db-973d-b7d93cfea71a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:20.397756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1178/2384] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:43:20 2022] POST /identity/v3/users => generated 290 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:43:20.401508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9cb47c-8fdf-47cb-9a1b-90c3e6847a08 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:20.411737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9cb47c-8fdf-47cb-9a1b-90c3e6847a08 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:20.417847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beae6c61-b3eb-4cd3-868a-13efac66e5aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:20.418481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beae6c61-b3eb-4cd3-868a-13efac66e5aa admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:20.418763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beae6c61-b3eb-4cd3-868a-13efac66e5aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:20.419032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beae6c61-b3eb-4cd3-868a-13efac66e5aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:20.419675 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beae6c61-b3eb-4cd3-868a-13efac66e5aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:20.421410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beae6c61-b3eb-4cd3-868a-13efac66e5aa admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:20.435607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1207/2385] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:43:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:43:20.436910 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4aa1bd-d558-43db-973d-b7d93cfea71a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:20.447439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4aa1bd-d558-43db-973d-b7d93cfea71a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:20.453911 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e49b62-44db-4037-a50a-c7fc3d858a75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:20.454348 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e49b62-44db-4037-a50a-c7fc3d858a75 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:20.454640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e49b62-44db-4037-a50a-c7fc3d858a75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:20.454898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e49b62-44db-4037-a50a-c7fc3d858a75 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:20.455541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e49b62-44db-4037-a50a-c7fc3d858a75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:20.457177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e49b62-44db-4037-a50a-c7fc3d858a75 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:20.468942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1179/2386] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:43:20 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:43:20.473167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beae6c61-b3eb-4cd3-868a-13efac66e5aa admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:20.489752 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beae6c61-b3eb-4cd3-868a-13efac66e5aa admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:20.497391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40ca6ea-9296-4f98-b072-11a2af6395cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:20.498064 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40ca6ea-9296-4f98-b072-11a2af6395cd admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:20.498376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40ca6ea-9296-4f98-b072-11a2af6395cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:20.498617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40ca6ea-9296-4f98-b072-11a2af6395cd admin admin] PATH_INFO: `/v3/projects/74f642298efd4c0aa7b34eee43b5781a/users/5d2ee206adf54060877131b4bf12c893/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:20.515200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40ca6ea-9296-4f98-b072-11a2af6395cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=74f642298efd4c0aa7b34eee43b5781a, user_id=5d2ee206adf54060877131b4bf12c893, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:20.517910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40ca6ea-9296-4f98-b072-11a2af6395cd admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:20.533216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1208/2387] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:43:20 2022] PUT /identity/v3/projects/74f642298efd4c0aa7b34eee43b5781a/users/5d2ee206adf54060877131b4bf12c893/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:43:20.543557 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe38398-5c3d-4f8f-b85a-b4578f39a7b7 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:20.543845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe38398-5c3d-4f8f-b85a-b4578f39a7b7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:20.544122 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe38398-5c3d-4f8f-b85a-b4578f39a7b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:20.576269 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-7fe38398-5c3d-4f8f-b85a-b4578f39a7b7 None None] MFA Rules not processed for user `5d2ee206adf54060877131b4bf12c893`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:43:20.577511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-7fe38398-5c3d-4f8f-b85a-b4578f39a7b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:20.648987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1180/2388] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:43:20 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:43:20.653605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40ca6ea-9296-4f98-b072-11a2af6395cd admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:20.728876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40ca6ea-9296-4f98-b072-11a2af6395cd admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:20.733839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51267f7f-4ed5-4fc9-9123-de42cf4b6c7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:20.734464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51267f7f-4ed5-4fc9-9123-de42cf4b6c7f admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:20.734749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51267f7f-4ed5-4fc9-9123-de42cf4b6c7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:20.735021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51267f7f-4ed5-4fc9-9123-de42cf4b6c7f admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:20.735660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51267f7f-4ed5-4fc9-9123-de42cf4b6c7f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:20.737601 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51267f7f-4ed5-4fc9-9123-de42cf4b6c7f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:20.765587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1209/2389] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:43:20 2022] POST /identity/v3/projects => generated 379 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:43:20.769818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe38398-5c3d-4f8f-b85a-b4578f39a7b7 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:20.780810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe38398-5c3d-4f8f-b85a-b4578f39a7b7 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:20.810420 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:43:20.842083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da72db12-7565-4847-b7e2-372666a1e008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:20.842821 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da72db12-7565-4847-b7e2-372666a1e008 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:20.843142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da72db12-7565-4847-b7e2-372666a1e008 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:20.843422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da72db12-7565-4847-b7e2-372666a1e008 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:20.844120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da72db12-7565-4847-b7e2-372666a1e008 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:20.846229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da72db12-7565-4847-b7e2-372666a1e008 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:20.868809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1181/2390] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:43:20 2022] POST /identity/v3/users => generated 296 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:43:20.872827 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51267f7f-4ed5-4fc9-9123-de42cf4b6c7f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:20.884087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51267f7f-4ed5-4fc9-9123-de42cf4b6c7f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:20.888852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3baf7553-3768-470b-841c-623a414abf36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:20.889450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baf7553-3768-470b-841c-623a414abf36 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:20.889727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baf7553-3768-470b-841c-623a414abf36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:20.890078 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baf7553-3768-470b-841c-623a414abf36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:20.890796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3baf7553-3768-470b-841c-623a414abf36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:20.892466 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3baf7553-3768-470b-841c-623a414abf36 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:20.903070 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1210/2391] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:43:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:43:20.907303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da72db12-7565-4847-b7e2-372666a1e008 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:20.919540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da72db12-7565-4847-b7e2-372666a1e008 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:20.924537 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16a1be9-3ca2-4d34-8725-57239bd72731 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:20.925184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16a1be9-3ca2-4d34-8725-57239bd72731 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:20.925460 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16a1be9-3ca2-4d34-8725-57239bd72731 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:20.925730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16a1be9-3ca2-4d34-8725-57239bd72731 admin admin] PATH_INFO: `/v3/projects/bda425b125df4069a51181100cfc2a26/users/b7dc4266ebf14ca99cad600fd37ba8df/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:20.941031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16a1be9-3ca2-4d34-8725-57239bd72731 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bda425b125df4069a51181100cfc2a26, user_id=b7dc4266ebf14ca99cad600fd37ba8df, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:20.945249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16a1be9-3ca2-4d34-8725-57239bd72731 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:20.959772 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1182/2392] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:43:20 2022] PUT /identity/v3/projects/bda425b125df4069a51181100cfc2a26/users/b7dc4266ebf14ca99cad600fd37ba8df/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:43:20.963723 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3baf7553-3768-470b-841c-623a414abf36 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:20.975880 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3baf7553-3768-470b-841c-623a414abf36 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:21.056759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c150c332-5f57-4ca7-b31d-2259a22a977e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:21.057458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c150c332-5f57-4ca7-b31d-2259a22a977e admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:21.057804 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c150c332-5f57-4ca7-b31d-2259a22a977e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:21.058141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c150c332-5f57-4ca7-b31d-2259a22a977e admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:21.058796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c150c332-5f57-4ca7-b31d-2259a22a977e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:21.060414 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c150c332-5f57-4ca7-b31d-2259a22a977e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:21.074303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1211/2393] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:43:20 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:43:21.078586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16a1be9-3ca2-4d34-8725-57239bd72731 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:21.090755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16a1be9-3ca2-4d34-8725-57239bd72731 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:21.096020 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e521b659-cc79-4ac5-bbbb-4b89d92d2afc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:21.096528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e521b659-cc79-4ac5-bbbb-4b89d92d2afc admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:21.096807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e521b659-cc79-4ac5-bbbb-4b89d92d2afc admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:21.097106 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e521b659-cc79-4ac5-bbbb-4b89d92d2afc admin admin] PATH_INFO: `/v3/projects/bda425b125df4069a51181100cfc2a26/users/b7dc4266ebf14ca99cad600fd37ba8df/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:21.099424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e521b659-cc79-4ac5-bbbb-4b89d92d2afc admin admin] RBAC: Authorizing `identity:create_grant(project_id=bda425b125df4069a51181100cfc2a26, user_id=b7dc4266ebf14ca99cad600fd37ba8df, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:21.102735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e521b659-cc79-4ac5-bbbb-4b89d92d2afc admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:21.117487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1183/2394] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:43:21 2022] PUT /identity/v3/projects/bda425b125df4069a51181100cfc2a26/users/b7dc4266ebf14ca99cad600fd37ba8df/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:43:21.128086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd0167e-7099-48f3-aa85-58d5858f8335 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:21.128410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd0167e-7099-48f3-aa85-58d5858f8335 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:21.128692 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd0167e-7099-48f3-aa85-58d5858f8335 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:21.174904 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-3fd0167e-7099-48f3-aa85-58d5858f8335 None None] MFA Rules not processed for user `b7dc4266ebf14ca99cad600fd37ba8df`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:43:21.176148 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-3fd0167e-7099-48f3-aa85-58d5858f8335 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:21.207881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:43:21.261178 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1212/2395] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:43:21 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:43:21.272426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e521b659-cc79-4ac5-bbbb-4b89d92d2afc admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:21.274955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e521b659-cc79-4ac5-bbbb-4b89d92d2afc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:43:21.332273 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e521b659-cc79-4ac5-bbbb-4b89d92d2afc admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:21.336841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4b846a-f87e-4941-a796-fad537982c0a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:21.337442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4b846a-f87e-4941-a796-fad537982c0a service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:21.337719 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4b846a-f87e-4941-a796-fad537982c0a service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:21.337990 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4b846a-f87e-4941-a796-fad537982c0a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:21.347273 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4b846a-f87e-4941-a796-fad537982c0a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:21.348740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4b846a-f87e-4941-a796-fad537982c0a service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:21.360162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1184/2396] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:43:21 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:43:21.404432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd0167e-7099-48f3-aa85-58d5858f8335 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:43:21.414289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd0167e-7099-48f3-aa85-58d5858f8335 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:43:21.418013 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4db16-188f-4cb5-9db2-1f87c191d8dc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:43:21.418684 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4db16-188f-4cb5-9db2-1f87c191d8dc service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:43:21.419029 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4db16-188f-4cb5-9db2-1f87c191d8dc service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:43:21.419369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4db16-188f-4cb5-9db2-1f87c191d8dc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:43:21.490939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f4db16-188f-4cb5-9db2-1f87c191d8dc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:43:21.490939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f4db16-188f-4cb5-9db2-1f87c191d8dc service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:43:21.499746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1213/2397] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:43:21 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:43:31.354542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=56,keystone:INSERT=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:43:31.495577 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=48 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:44:08.701000 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4b846a-f87e-4941-a796-fad537982c0a service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:44:08.702390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-aa4b846a-f87e-4941-a796-fad537982c0a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:44:08.806484 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4b846a-f87e-4941-a796-fad537982c0a service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:44:08.811500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e33b19c-2fa6-4af3-8cba-3a2b9d535ec7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:44:08.812093 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e33b19c-2fa6-4af3-8cba-3a2b9d535ec7 service placement] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:44:08.812409 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e33b19c-2fa6-4af3-8cba-3a2b9d535ec7 service placement] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:44:08.812683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e33b19c-2fa6-4af3-8cba-3a2b9d535ec7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:44:08.814397 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0e33b19c-2fa6-4af3-8cba-3a2b9d535ec7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:44:08.825946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e33b19c-2fa6-4af3-8cba-3a2b9d535ec7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:44:08.827520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e33b19c-2fa6-4af3-8cba-3a2b9d535ec7 service placement] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:44:08.837620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1185/2398] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:44:08 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:44:18.832988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=48 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:46:16.709498 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4db16-188f-4cb5-9db2-1f87c191d8dc service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:46:16.737849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4db16-188f-4cb5-9db2-1f87c191d8dc service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:46:16.741409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f6e355-7fcc-4fa9-86ac-4ad02099b453 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:46:16.741998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f6e355-7fcc-4fa9-86ac-4ad02099b453 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:46:16.742275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f6e355-7fcc-4fa9-86ac-4ad02099b453 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:46:16.742545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f6e355-7fcc-4fa9-86ac-4ad02099b453 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:46:16.744571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-31f6e355-7fcc-4fa9-86ac-4ad02099b453 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:46:16.847217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f6e355-7fcc-4fa9-86ac-4ad02099b453 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:46:16.848668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f6e355-7fcc-4fa9-86ac-4ad02099b453 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:46:16.866434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1214/2399] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:46:16 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:46:16.910708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e33b19c-2fa6-4af3-8cba-3a2b9d535ec7 service placement] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:46:16.929426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e33b19c-2fa6-4af3-8cba-3a2b9d535ec7 service placement] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:46:16.933070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d18a07c-70c6-49b8-a6f8-f22283e478d4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:46:16.934160 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d18a07c-70c6-49b8-a6f8-f22283e478d4 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:46:16.934437 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d18a07c-70c6-49b8-a6f8-f22283e478d4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:46:16.934706 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d18a07c-70c6-49b8-a6f8-f22283e478d4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:46:16.936490 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3d18a07c-70c6-49b8-a6f8-f22283e478d4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:46:17.016553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d18a07c-70c6-49b8-a6f8-f22283e478d4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:46:17.017943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d18a07c-70c6-49b8-a6f8-f22283e478d4 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:46:17.036247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1186/2400] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:46:16 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:46:17.080191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f6e355-7fcc-4fa9-86ac-4ad02099b453 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:46:17.088732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f6e355-7fcc-4fa9-86ac-4ad02099b453 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:46:17.092498 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15e7030-79bb-4946-97dd-b6512717409d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:46:17.093326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15e7030-79bb-4946-97dd-b6512717409d service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:46:17.093610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15e7030-79bb-4946-97dd-b6512717409d service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:46:17.093881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15e7030-79bb-4946-97dd-b6512717409d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:46:17.095589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a15e7030-79bb-4946-97dd-b6512717409d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:46:17.173766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15e7030-79bb-4946-97dd-b6512717409d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:46:17.175180 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15e7030-79bb-4946-97dd-b6512717409d service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:46:17.189062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:46:17.194253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1215/2401] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:46:17 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:46:27.033626 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=46 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:46:27.192797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:47:17.199037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d18a07c-70c6-49b8-a6f8-f22283e478d4 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:17.270747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d18a07c-70c6-49b8-a6f8-f22283e478d4 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:17.274528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce9b837-05ea-4d35-95a1-52af15fe9f51 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:17.275124 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce9b837-05ea-4d35-95a1-52af15fe9f51 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:17.275425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce9b837-05ea-4d35-95a1-52af15fe9f51 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:17.275705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce9b837-05ea-4d35-95a1-52af15fe9f51 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:17.277485 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5ce9b837-05ea-4d35-95a1-52af15fe9f51 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:17.316575 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15e7030-79bb-4946-97dd-b6512717409d service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:17.325921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15e7030-79bb-4946-97dd-b6512717409d service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:17.329513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87a488f-85c5-42df-b354-ee42d8e5c74c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:17.330107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87a488f-85c5-42df-b354-ee42d8e5c74c service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:17.330413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87a488f-85c5-42df-b354-ee42d8e5c74c service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:17.330685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87a488f-85c5-42df-b354-ee42d8e5c74c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:17.332478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e87a488f-85c5-42df-b354-ee42d8e5c74c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:17.345963 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce9b837-05ea-4d35-95a1-52af15fe9f51 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:17.347406 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce9b837-05ea-4d35-95a1-52af15fe9f51 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:17.357262 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1187/2402] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:47:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:47:17.396044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87a488f-85c5-42df-b354-ee42d8e5c74c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:17.397513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87a488f-85c5-42df-b354-ee42d8e5c74c service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:17.407026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1216/2403] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:47:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:47:23.024605 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce9b837-05ea-4d35-95a1-52af15fe9f51 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:23.036425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce9b837-05ea-4d35-95a1-52af15fe9f51 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:23.041380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2b3f89-b52a-46be-ad6f-90d14c9a32ca service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:23.041975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2b3f89-b52a-46be-ad6f-90d14c9a32ca service placement] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.042250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2b3f89-b52a-46be-ad6f-90d14c9a32ca service placement] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.042518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2b3f89-b52a-46be-ad6f-90d14c9a32ca service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.044219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-ca2b3f89-b52a-46be-ad6f-90d14c9a32ca service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:23.056190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2b3f89-b52a-46be-ad6f-90d14c9a32ca service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:23.057803 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2b3f89-b52a-46be-ad6f-90d14c9a32ca service placement] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:23.068060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1188/2404] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:47:23 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:47:23.139412 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87a488f-85c5-42df-b354-ee42d8e5c74c service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:23.141004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e87a488f-85c5-42df-b354-ee42d8e5c74c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:23.153085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2b3f89-b52a-46be-ad6f-90d14c9a32ca service placement] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:23.154704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-ca2b3f89-b52a-46be-ad6f-90d14c9a32ca service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:23.192216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87a488f-85c5-42df-b354-ee42d8e5c74c service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:23.194262 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2b3f89-b52a-46be-ad6f-90d14c9a32ca service placement] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:23.196932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d65473-1c2f-41ee-a436-937b62c8bb0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:23.197555 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d65473-1c2f-41ee-a436-937b62c8bb0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.197839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d65473-1c2f-41ee-a436-937b62c8bb0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.198111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d65473-1c2f-41ee-a436-937b62c8bb0b admin admin] PATH_INFO: `/v3/users/fdb0f345d2e04483b5166e1eb728bf91` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.199234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d65473-1c2f-41ee-a436-937b62c8bb0b admin admin] RBAC: Authorizing `identity:delete_user(user_id=fdb0f345d2e04483b5166e1eb728bf91)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:23.199337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0802c9-1df6-4fd1-a500-4db3c189c896 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:23.200145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0802c9-1df6-4fd1-a500-4db3c189c896 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.200522 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0802c9-1df6-4fd1-a500-4db3c189c896 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.200868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0802c9-1df6-4fd1-a500-4db3c189c896 admin admin] PATH_INFO: `/v3/users/820f4bb963644499a77fa1f3edd59d08` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.201174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d65473-1c2f-41ee-a436-937b62c8bb0b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:23.202129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0802c9-1df6-4fd1-a500-4db3c189c896 admin admin] RBAC: Authorizing `identity:delete_user(user_id=820f4bb963644499a77fa1f3edd59d08)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:23.204102 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0802c9-1df6-4fd1-a500-4db3c189c896 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:23.250868 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-34d65473-1c2f-41ee-a436-937b62c8bb0b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fdb0f345d2e04483b5166e1eb728bf91'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.251611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-34d65473-1c2f-41ee-a436-937b62c8bb0b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fdb0f345d2e04483b5166e1eb728bf91'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.260665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-34d65473-1c2f-41ee-a436-937b62c8bb0b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fdb0f345d2e04483b5166e1eb728bf91'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.260979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0d0802c9-1df6-4fd1-a500-4db3c189c896 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '820f4bb963644499a77fa1f3edd59d08'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.261576 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-34d65473-1c2f-41ee-a436-937b62c8bb0b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fdb0f345d2e04483b5166e1eb728bf91'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.261697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0d0802c9-1df6-4fd1-a500-4db3c189c896 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '820f4bb963644499a77fa1f3edd59d08'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.262265 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0d0802c9-1df6-4fd1-a500-4db3c189c896 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '820f4bb963644499a77fa1f3edd59d08'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.270738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0d0802c9-1df6-4fd1-a500-4db3c189c896 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '820f4bb963644499a77fa1f3edd59d08'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.276300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:47:23.285110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0d0802c9-1df6-4fd1-a500-4db3c189c896 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '820f4bb963644499a77fa1f3edd59d08'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.285813 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-34d65473-1c2f-41ee-a436-937b62c8bb0b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fdb0f345d2e04483b5166e1eb728bf91'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.299339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1189/2405] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:47:23 2022] DELETE /identity/v3/users/820f4bb963644499a77fa1f3edd59d08 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:47:23.303286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1217/2406] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:47:23 2022] DELETE /identity/v3/users/fdb0f345d2e04483b5166e1eb728bf91 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:47:23.487252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0802c9-1df6-4fd1-a500-4db3c189c896 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:23.489730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d65473-1c2f-41ee-a436-937b62c8bb0b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:23.492264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-34d65473-1c2f-41ee-a436-937b62c8bb0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:23.494523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0d0802c9-1df6-4fd1-a500-4db3c189c896 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:23.547608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:47:23.568953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0802c9-1df6-4fd1-a500-4db3c189c896 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:23.573445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d65473-1c2f-41ee-a436-937b62c8bb0b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:23.574202 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:23.574834 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.575112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.575386 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e admin admin] PATH_INFO: `/v3/projects/f7f6e096c5a54a7ebc843b012e82e0f4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.576535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7f6e096c5a54a7ebc843b012e82e0f4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:23.578317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739d8259-663f-47e3-b8d6-c7a76e50d23e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:23.578442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:23.578931 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d8259-663f-47e3-b8d6-c7a76e50d23e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.579218 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d8259-663f-47e3-b8d6-c7a76e50d23e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.579489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d8259-663f-47e3-b8d6-c7a76e50d23e admin admin] PATH_INFO: `/v3/projects/ba288a6bdc4e41729bd679733eaee61e` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.580865 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739d8259-663f-47e3-b8d6-c7a76e50d23e admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba288a6bdc4e41729bd679733eaee61e)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:23.582769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739d8259-663f-47e3-b8d6-c7a76e50d23e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:23.631728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7f6e096c5a54a7ebc843b012e82e0f4', 'request_id': 'req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.637370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-739d8259-663f-47e3-b8d6-c7a76e50d23e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba288a6bdc4e41729bd679733eaee61e', 'request_id': 'req-739d8259-663f-47e3-b8d6-c7a76e50d23e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.638140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7f6e096c5a54a7ebc843b012e82e0f4', 'request_id': 'req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.645010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-739d8259-663f-47e3-b8d6-c7a76e50d23e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba288a6bdc4e41729bd679733eaee61e', 'request_id': 'req-739d8259-663f-47e3-b8d6-c7a76e50d23e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.648859 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7f6e096c5a54a7ebc843b012e82e0f4', 'request_id': 'req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.653666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-739d8259-663f-47e3-b8d6-c7a76e50d23e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba288a6bdc4e41729bd679733eaee61e', 'request_id': 'req-739d8259-663f-47e3-b8d6-c7a76e50d23e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.668256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-739d8259-663f-47e3-b8d6-c7a76e50d23e admin admin] The token cache is being invalidate because project ba288a6bdc4e41729bd679733eaee61e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:47:23.668750 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-739d8259-663f-47e3-b8d6-c7a76e50d23e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.669577 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-739d8259-663f-47e3-b8d6-c7a76e50d23e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.671207 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1218/2407] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:47:23 2022] DELETE /identity/v3/projects/ba288a6bdc4e41729bd679733eaee61e => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:47:23.672561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e admin admin] The token cache is being invalidate because project f7f6e096c5a54a7ebc843b012e82e0f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:47:23.672882 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.674673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-351fecd4-75db-4bfc-81ab-15c28b3c7c2e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:23.675832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1190/2408] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:47:23 2022] DELETE /identity/v3/projects/f7f6e096c5a54a7ebc843b012e82e0f4 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:47:23.677502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f869e4fd-2e69-40a5-843d-d34dfd9965f2 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.677787 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f869e4fd-2e69-40a5-843d-d34dfd9965f2 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.678061 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f869e4fd-2e69-40a5-843d-d34dfd9965f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.681273 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3002abd-e06b-4532-b4f0-b90199cf767a None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.681562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3002abd-e06b-4532-b4f0-b90199cf767a None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.681842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3002abd-e06b-4532-b4f0-b90199cf767a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.701705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-f3002abd-e06b-4532-b4f0-b90199cf767a None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:47:23.702818 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-f869e4fd-2e69-40a5-843d-d34dfd9965f2 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:47:23.702954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f3002abd-e06b-4532-b4f0-b90199cf767a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:23.704000 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f869e4fd-2e69-40a5-843d-d34dfd9965f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:23.761224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1191/2409] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:47:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:47:23.763573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1219/2410] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:47:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:47:23.771188 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff58f1b-44b1-4a93-a4f2-843aeb056145 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.771583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff58f1b-44b1-4a93-a4f2-843aeb056145 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.772234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff58f1b-44b1-4a93-a4f2-843aeb056145 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.772549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974b6afb-253b-47a1-9ff4-96cd1c1d6d27 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.772831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974b6afb-253b-47a1-9ff4-96cd1c1d6d27 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.773105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974b6afb-253b-47a1-9ff4-96cd1c1d6d27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.793271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-974b6afb-253b-47a1-9ff4-96cd1c1d6d27 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:47:23.794415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-bff58f1b-44b1-4a93-a4f2-843aeb056145 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:47:23.794551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-974b6afb-253b-47a1-9ff4-96cd1c1d6d27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:23.795589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-bff58f1b-44b1-4a93-a4f2-843aeb056145 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:23.803278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1220/2411] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:47:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:47:23.804707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1192/2412] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:47:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:47:23.807092 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974b6afb-253b-47a1-9ff4-96cd1c1d6d27 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:23.808566 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff58f1b-44b1-4a93-a4f2-843aeb056145 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:23.813889 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:47:23.816832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974b6afb-253b-47a1-9ff4-96cd1c1d6d27 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:23.817691 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff58f1b-44b1-4a93-a4f2-843aeb056145 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:23.823143 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee01bb44-17d3-454a-92d8-d9e272bd1769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:23.823968 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45ed78e-4f01-46f8-b566-d3f1bbfece26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:23.824090 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee01bb44-17d3-454a-92d8-d9e272bd1769 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.824436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee01bb44-17d3-454a-92d8-d9e272bd1769 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.824603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45ed78e-4f01-46f8-b566-d3f1bbfece26 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.824720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee01bb44-17d3-454a-92d8-d9e272bd1769 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.825282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45ed78e-4f01-46f8-b566-d3f1bbfece26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.825392 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee01bb44-17d3-454a-92d8-d9e272bd1769 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:47:23.825561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45ed78e-4f01-46f8-b566-d3f1bbfece26 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.825663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee01bb44-17d3-454a-92d8-d9e272bd1769 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:23.826477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45ed78e-4f01-46f8-b566-d3f1bbfece26 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:47:23.827064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45ed78e-4f01-46f8-b566-d3f1bbfece26 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:23.827209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee01bb44-17d3-454a-92d8-d9e272bd1769 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:23.829198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45ed78e-4f01-46f8-b566-d3f1bbfece26 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:23.839396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1221/2413] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:47:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:47:23.844341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1193/2414] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:47:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:47:23.847104 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee01bb44-17d3-454a-92d8-d9e272bd1769 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:23.847252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45ed78e-4f01-46f8-b566-d3f1bbfece26 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:23.856934 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee01bb44-17d3-454a-92d8-d9e272bd1769 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:23.859606 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45ed78e-4f01-46f8-b566-d3f1bbfece26 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:23.861873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff622dd-25bd-42ce-ba16-501dec840ea0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:23.862467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff622dd-25bd-42ce-ba16-501dec840ea0 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.862745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff622dd-25bd-42ce-ba16-501dec840ea0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.863017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff622dd-25bd-42ce-ba16-501dec840ea0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.863774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff622dd-25bd-42ce-ba16-501dec840ea0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:23.864673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a64f37c-3422-4796-b527-d3f0bca58a2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:23.865258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a64f37c-3422-4796-b527-d3f0bca58a2c admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.865539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a64f37c-3422-4796-b527-d3f0bca58a2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.865817 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a64f37c-3422-4796-b527-d3f0bca58a2c admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.865916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff622dd-25bd-42ce-ba16-501dec840ea0 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:23.866513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a64f37c-3422-4796-b527-d3f0bca58a2c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:23.868794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a64f37c-3422-4796-b527-d3f0bca58a2c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:23.895226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1222/2415] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:47:23 2022] POST /identity/v3/projects => generated 371 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:47:23.901123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1194/2416] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:47:23 2022] POST /identity/v3/projects => generated 375 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:47:23.901288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff622dd-25bd-42ce-ba16-501dec840ea0 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:23.902404 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a64f37c-3422-4796-b527-d3f0bca58a2c admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:23.933910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:47:23.968163 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a64f37c-3422-4796-b527-d3f0bca58a2c admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:23.972210 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff622dd-25bd-42ce-ba16-501dec840ea0 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:23.973638 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1664ed35-0a04-4f48-bbb8-289693bd75a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:23.974264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1664ed35-0a04-4f48-bbb8-289693bd75a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.974549 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1664ed35-0a04-4f48-bbb8-289693bd75a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.974823 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1664ed35-0a04-4f48-bbb8-289693bd75a5 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.975499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1664ed35-0a04-4f48-bbb8-289693bd75a5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:23.977083 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba900ebb-9030-4968-bd30-28d0d526e27b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:23.977453 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1664ed35-0a04-4f48-bbb8-289693bd75a5 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:23.977666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba900ebb-9030-4968-bd30-28d0d526e27b admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:23.977945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba900ebb-9030-4968-bd30-28d0d526e27b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:23.978217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba900ebb-9030-4968-bd30-28d0d526e27b admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:23.978839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba900ebb-9030-4968-bd30-28d0d526e27b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:23.980741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba900ebb-9030-4968-bd30-28d0d526e27b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:24.001587 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1195/2417] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:47:23 2022] POST /identity/v3/users => generated 288 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:47:24.004531 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1223/2418] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:47:23 2022] POST /identity/v3/users => generated 286 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:47:24.006864 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba900ebb-9030-4968-bd30-28d0d526e27b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:24.007040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1664ed35-0a04-4f48-bbb8-289693bd75a5 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:24.016653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba900ebb-9030-4968-bd30-28d0d526e27b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:24.018028 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1664ed35-0a04-4f48-bbb8-289693bd75a5 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:24.021811 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c412240-30cb-4e5d-b812-e88163759ec8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:24.022419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c412240-30cb-4e5d-b812-e88163759ec8 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:24.022700 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c412240-30cb-4e5d-b812-e88163759ec8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:24.022861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc962b7-0a82-44ea-b576-e06e715b6528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:24.022992 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c412240-30cb-4e5d-b812-e88163759ec8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:24.023580 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc962b7-0a82-44ea-b576-e06e715b6528 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:24.023686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c412240-30cb-4e5d-b812-e88163759ec8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:24.023985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc962b7-0a82-44ea-b576-e06e715b6528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:24.024445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc962b7-0a82-44ea-b576-e06e715b6528 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:24.025088 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc962b7-0a82-44ea-b576-e06e715b6528 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:24.025728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c412240-30cb-4e5d-b812-e88163759ec8 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:24.026674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc962b7-0a82-44ea-b576-e06e715b6528 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:24.035637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1224/2419] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:47:24 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:47:24.037937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1196/2420] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:47:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:47:24.039482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c412240-30cb-4e5d-b812-e88163759ec8 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:24.041509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc962b7-0a82-44ea-b576-e06e715b6528 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:24.049174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c412240-30cb-4e5d-b812-e88163759ec8 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:24.052020 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc962b7-0a82-44ea-b576-e06e715b6528 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:24.053924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fb5a06-1a26-49eb-bfc5-aed06d83ddf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:24.054536 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb5a06-1a26-49eb-bfc5-aed06d83ddf5 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:24.054873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb5a06-1a26-49eb-bfc5-aed06d83ddf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:24.055146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb5a06-1a26-49eb-bfc5-aed06d83ddf5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:24.055814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fb5a06-1a26-49eb-bfc5-aed06d83ddf5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:24.056733 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcfda89-71a1-419f-b157-1c720b8df741 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:24.057309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcfda89-71a1-419f-b157-1c720b8df741 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:24.057485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fb5a06-1a26-49eb-bfc5-aed06d83ddf5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:24.057616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcfda89-71a1-419f-b157-1c720b8df741 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:24.057937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcfda89-71a1-419f-b157-1c720b8df741 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:24.058811 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcfda89-71a1-419f-b157-1c720b8df741 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:24.060690 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcfda89-71a1-419f-b157-1c720b8df741 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:24.068149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1225/2421] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:47:24 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:47:24.071928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1197/2422] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:47:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:47:24.072227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fb5a06-1a26-49eb-bfc5-aed06d83ddf5 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:24.075877 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcfda89-71a1-419f-b157-1c720b8df741 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:24.082252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fb5a06-1a26-49eb-bfc5-aed06d83ddf5 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:24.086366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcfda89-71a1-419f-b157-1c720b8df741 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:24.086820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f241e381-0995-4dc6-bb78-814b006ccbac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:24.087441 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f241e381-0995-4dc6-bb78-814b006ccbac admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:24.087718 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f241e381-0995-4dc6-bb78-814b006ccbac admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:24.087988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f241e381-0995-4dc6-bb78-814b006ccbac admin admin] PATH_INFO: `/v3/projects/557063c52e344d5383e2ceabda9bcac7/users/9ef1cdec1a5a4c4ea1de2308d0f8de31/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:24.091413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8077286-475d-46f4-a92f-abff6f396431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:24.092144 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8077286-475d-46f4-a92f-abff6f396431 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:24.092524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8077286-475d-46f4-a92f-abff6f396431 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:24.092815 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8077286-475d-46f4-a92f-abff6f396431 admin admin] PATH_INFO: `/v3/projects/7709e0f0fe264af1a74065f0d38781fd/users/5cb3434ff57e4764b70129ad7902186d/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:24.104460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f241e381-0995-4dc6-bb78-814b006ccbac admin admin] RBAC: Authorizing `identity:create_grant(project_id=557063c52e344d5383e2ceabda9bcac7, user_id=9ef1cdec1a5a4c4ea1de2308d0f8de31, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:24.107238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f241e381-0995-4dc6-bb78-814b006ccbac admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:24.108341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8077286-475d-46f4-a92f-abff6f396431 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7709e0f0fe264af1a74065f0d38781fd, user_id=5cb3434ff57e4764b70129ad7902186d, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:24.111261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8077286-475d-46f4-a92f-abff6f396431 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:24.121103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1226/2423] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:47:24 2022] PUT /identity/v3/projects/557063c52e344d5383e2ceabda9bcac7/users/9ef1cdec1a5a4c4ea1de2308d0f8de31/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:47:24.129216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1198/2424] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:47:24 2022] PUT /identity/v3/projects/7709e0f0fe264af1a74065f0d38781fd/users/5cb3434ff57e4764b70129ad7902186d/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:47:24.132631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a39050-4f07-4039-91fc-87e9c36d29ea None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:24.133043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a39050-4f07-4039-91fc-87e9c36d29ea None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:24.133386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a39050-4f07-4039-91fc-87e9c36d29ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:24.139471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848d453c-406c-4f89-81e3-a8b4ec47b758 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:24.139881 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848d453c-406c-4f89-81e3-a8b4ec47b758 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:24.140184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848d453c-406c-4f89-81e3-a8b4ec47b758 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:24.162591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-12a39050-4f07-4039-91fc-87e9c36d29ea None None] MFA Rules not processed for user `9ef1cdec1a5a4c4ea1de2308d0f8de31`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:47:24.163804 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-12a39050-4f07-4039-91fc-87e9c36d29ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:24.169230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-848d453c-406c-4f89-81e3-a8b4ec47b758 None None] MFA Rules not processed for user `5cb3434ff57e4764b70129ad7902186d`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:47:24.170427 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-848d453c-406c-4f89-81e3-a8b4ec47b758 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:24.230760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1227/2425] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:47:24 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:47:24.233808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1199/2426] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:47:24 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:47:24.245720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a39050-4f07-4039-91fc-87e9c36d29ea None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:24.246987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-12a39050-4f07-4039-91fc-87e9c36d29ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:24.248191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848d453c-406c-4f89-81e3-a8b4ec47b758 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:24.250108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-848d453c-406c-4f89-81e3-a8b4ec47b758 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:24.255969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:47:24.315766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:47:24.369094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848d453c-406c-4f89-81e3-a8b4ec47b758 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:24.373639 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a39050-4f07-4039-91fc-87e9c36d29ea None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:24.375938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc74fa6-b025-4a19-b274-c368e8a0b131 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:24.376598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc74fa6-b025-4a19-b274-c368e8a0b131 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:24.377385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc74fa6-b025-4a19-b274-c368e8a0b131 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:24.377650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc74fa6-b025-4a19-b274-c368e8a0b131 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:24.386906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac28e58-438c-4e1a-910b-84763ef9e679 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:24.387369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac28e58-438c-4e1a-910b-84763ef9e679 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:24.387644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac28e58-438c-4e1a-910b-84763ef9e679 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:24.387917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac28e58-438c-4e1a-910b-84763ef9e679 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:24.389138 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc74fa6-b025-4a19-b274-c368e8a0b131 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:24.390738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc74fa6-b025-4a19-b274-c368e8a0b131 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:24.396781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac28e58-438c-4e1a-910b-84763ef9e679 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:24.398395 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac28e58-438c-4e1a-910b-84763ef9e679 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:24.402116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1200/2427] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:47:24 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:47:24.410655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1228/2428] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:47:24 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:47:24.852158 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc74fa6-b025-4a19-b274-c368e8a0b131 service nova] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:24.917533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc74fa6-b025-4a19-b274-c368e8a0b131 service nova] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:24.922632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9cc4ba-6fef-4b02-8e7e-c1d38c8bf636 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:24.923360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9cc4ba-6fef-4b02-8e7e-c1d38c8bf636 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:24.923822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9cc4ba-6fef-4b02-8e7e-c1d38c8bf636 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:24.924193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9cc4ba-6fef-4b02-8e7e-c1d38c8bf636 admin admin] PATH_INFO: `/v3/users/5cb3434ff57e4764b70129ad7902186d` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:24.925444 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9cc4ba-6fef-4b02-8e7e-c1d38c8bf636 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5cb3434ff57e4764b70129ad7902186d)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:24.927508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9cc4ba-6fef-4b02-8e7e-c1d38c8bf636 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:24.975369 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1c9cc4ba-6fef-4b02-8e7e-c1d38c8bf636 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5cb3434ff57e4764b70129ad7902186d'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:24.975948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1c9cc4ba-6fef-4b02-8e7e-c1d38c8bf636 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5cb3434ff57e4764b70129ad7902186d'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:24.976591 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1c9cc4ba-6fef-4b02-8e7e-c1d38c8bf636 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5cb3434ff57e4764b70129ad7902186d'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:24.983902 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1c9cc4ba-6fef-4b02-8e7e-c1d38c8bf636 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5cb3434ff57e4764b70129ad7902186d'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:24.989433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:47:24.998328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1c9cc4ba-6fef-4b02-8e7e-c1d38c8bf636 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5cb3434ff57e4764b70129ad7902186d'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:25.011134 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1201/2429] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:47:24 2022] DELETE /identity/v3/users/5cb3434ff57e4764b70129ad7902186d => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:47:25.022470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328b54ff-b8f9-4bc9-b539-9898f4b15ce1 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:25.022753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328b54ff-b8f9-4bc9-b539-9898f4b15ce1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:25.023293 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328b54ff-b8f9-4bc9-b539-9898f4b15ce1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:25.041892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-328b54ff-b8f9-4bc9-b539-9898f4b15ce1 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:47:25.043099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-328b54ff-b8f9-4bc9-b539-9898f4b15ce1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:25.101386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1229/2430] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:47:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:47:25.112117 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9cc4ba-6fef-4b02-8e7e-c1d38c8bf636 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:25.113559 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1c9cc4ba-6fef-4b02-8e7e-c1d38c8bf636 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:25.184331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9cc4ba-6fef-4b02-8e7e-c1d38c8bf636 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:25.187990 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887c53f2-3141-4c0c-b8c2-0fe16145a365 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:25.188652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c53f2-3141-4c0c-b8c2-0fe16145a365 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:25.188935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c53f2-3141-4c0c-b8c2-0fe16145a365 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:25.189209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c53f2-3141-4c0c-b8c2-0fe16145a365 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:25.198921 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887c53f2-3141-4c0c-b8c2-0fe16145a365 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:25.200384 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887c53f2-3141-4c0c-b8c2-0fe16145a365 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:25.210398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1202/2431] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:47:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:47:25.480121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328b54ff-b8f9-4bc9-b539-9898f4b15ce1 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:25.481651 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-328b54ff-b8f9-4bc9-b539-9898f4b15ce1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:25.494826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328b54ff-b8f9-4bc9-b539-9898f4b15ce1 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:25.499476 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3caad146-be8d-42fd-8137-1fe399751a15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:25.500114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caad146-be8d-42fd-8137-1fe399751a15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:25.500421 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caad146-be8d-42fd-8137-1fe399751a15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:25.500731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caad146-be8d-42fd-8137-1fe399751a15 admin admin] PATH_INFO: `/v3/projects/7709e0f0fe264af1a74065f0d38781fd` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:25.501857 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3caad146-be8d-42fd-8137-1fe399751a15 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7709e0f0fe264af1a74065f0d38781fd)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:25.504258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3caad146-be8d-42fd-8137-1fe399751a15 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:25.550623 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-3caad146-be8d-42fd-8137-1fe399751a15 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7709e0f0fe264af1a74065f0d38781fd', 'request_id': 'req-3caad146-be8d-42fd-8137-1fe399751a15'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:25.555988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-3caad146-be8d-42fd-8137-1fe399751a15 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7709e0f0fe264af1a74065f0d38781fd', 'request_id': 'req-3caad146-be8d-42fd-8137-1fe399751a15'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:25.563150 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-3caad146-be8d-42fd-8137-1fe399751a15 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7709e0f0fe264af1a74065f0d38781fd', 'request_id': 'req-3caad146-be8d-42fd-8137-1fe399751a15'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:25.574497 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-3caad146-be8d-42fd-8137-1fe399751a15 admin admin] The token cache is being invalidate because project 7709e0f0fe264af1a74065f0d38781fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:47:25.574788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-3caad146-be8d-42fd-8137-1fe399751a15 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:25.575408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-3caad146-be8d-42fd-8137-1fe399751a15 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:47:25.576899 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1230/2432] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:47:25 2022] DELETE /identity/v3/projects/7709e0f0fe264af1a74065f0d38781fd => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:47:25.581726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec99a9fd-73f2-49ac-b02a-d42b39ab0af4 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:25.582009 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec99a9fd-73f2-49ac-b02a-d42b39ab0af4 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:25.582281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec99a9fd-73f2-49ac-b02a-d42b39ab0af4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:25.640125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-ec99a9fd-73f2-49ac-b02a-d42b39ab0af4 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:47:25.641356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-ec99a9fd-73f2-49ac-b02a-d42b39ab0af4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:25.699024 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1203/2433] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:47:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:47:25.706931 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f2928c-f360-4b68-b259-765d9c42dcc1 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:25.707208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f2928c-f360-4b68-b259-765d9c42dcc1 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:25.707477 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f2928c-f360-4b68-b259-765d9c42dcc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:25.726821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-a2f2928c-f360-4b68-b259-765d9c42dcc1 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:47:25.727999 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a2f2928c-f360-4b68-b259-765d9c42dcc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:25.735851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1231/2434] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:47:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:47:25.740510 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec99a9fd-73f2-49ac-b02a-d42b39ab0af4 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:25.750063 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec99a9fd-73f2-49ac-b02a-d42b39ab0af4 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:25.754553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4131a83-16a0-4c4b-8da3-cce3ded71079 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:25.755134 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4131a83-16a0-4c4b-8da3-cce3ded71079 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:25.755410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4131a83-16a0-4c4b-8da3-cce3ded71079 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:25.755680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4131a83-16a0-4c4b-8da3-cce3ded71079 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:25.756330 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4131a83-16a0-4c4b-8da3-cce3ded71079 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:47:25.756637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4131a83-16a0-4c4b-8da3-cce3ded71079 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:25.758224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4131a83-16a0-4c4b-8da3-cce3ded71079 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:25.769514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1204/2435] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:47:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:47:25.773375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f2928c-f360-4b68-b259-765d9c42dcc1 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:25.782343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f2928c-f360-4b68-b259-765d9c42dcc1 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:25.787721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2992c705-afe8-478c-b820-66163878106b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:25.788329 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2992c705-afe8-478c-b820-66163878106b admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:25.788620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2992c705-afe8-478c-b820-66163878106b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:25.788895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2992c705-afe8-478c-b820-66163878106b admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:25.789540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2992c705-afe8-478c-b820-66163878106b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:25.791541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2992c705-afe8-478c-b820-66163878106b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:25.829548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1232/2436] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:47:25 2022] POST /identity/v3/projects => generated 383 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:47:25.833533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4131a83-16a0-4c4b-8da3-cce3ded71079 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:25.843448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4131a83-16a0-4c4b-8da3-cce3ded71079 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:25.870045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:47:25.901823 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae624da1-2ada-4e8c-b327-6ba9458a9ca3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:25.902424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae624da1-2ada-4e8c-b327-6ba9458a9ca3 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:25.902735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae624da1-2ada-4e8c-b327-6ba9458a9ca3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:25.903002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae624da1-2ada-4e8c-b327-6ba9458a9ca3 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:25.903656 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae624da1-2ada-4e8c-b327-6ba9458a9ca3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:25.905630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae624da1-2ada-4e8c-b327-6ba9458a9ca3 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:25.927390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1205/2437] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:47:25 2022] POST /identity/v3/users => generated 292 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:47:25.931508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2992c705-afe8-478c-b820-66163878106b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:25.941667 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2992c705-afe8-478c-b820-66163878106b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:25.946227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc0886e-d53e-4685-9441-c78a9df1f9cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:25.946810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc0886e-d53e-4685-9441-c78a9df1f9cb admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:25.947090 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc0886e-d53e-4685-9441-c78a9df1f9cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:25.947363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc0886e-d53e-4685-9441-c78a9df1f9cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:25.948003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc0886e-d53e-4685-9441-c78a9df1f9cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:25.949625 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc0886e-d53e-4685-9441-c78a9df1f9cb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:25.954707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:47:25.960400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1233/2438] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:47:25 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:47:25.963919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae624da1-2ada-4e8c-b327-6ba9458a9ca3 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:25.974320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae624da1-2ada-4e8c-b327-6ba9458a9ca3 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:25.978831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47959846-ffc1-4333-8e4c-90fb3758ea9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:25.979406 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47959846-ffc1-4333-8e4c-90fb3758ea9d admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:25.979679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47959846-ffc1-4333-8e4c-90fb3758ea9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:25.979945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47959846-ffc1-4333-8e4c-90fb3758ea9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:25.980605 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47959846-ffc1-4333-8e4c-90fb3758ea9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:25.982186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47959846-ffc1-4333-8e4c-90fb3758ea9d admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:25.993826 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1206/2439] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:47:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:47:25.997829 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc0886e-d53e-4685-9441-c78a9df1f9cb admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:26.007648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc0886e-d53e-4685-9441-c78a9df1f9cb admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:26.012228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b14329b-6046-479c-9e49-c6724f099386 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:26.012905 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b14329b-6046-479c-9e49-c6724f099386 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:26.013183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b14329b-6046-479c-9e49-c6724f099386 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:26.013483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b14329b-6046-479c-9e49-c6724f099386 admin admin] PATH_INFO: `/v3/projects/bcc2ff8cbc234c6d9917a05fd8b9b201/users/15db9d18e7b845f1838bc5f19d71b737/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:26.028578 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b14329b-6046-479c-9e49-c6724f099386 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bcc2ff8cbc234c6d9917a05fd8b9b201, user_id=15db9d18e7b845f1838bc5f19d71b737, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:26.031837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b14329b-6046-479c-9e49-c6724f099386 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:26.044416 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1234/2440] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:47:25 2022] PUT /identity/v3/projects/bcc2ff8cbc234c6d9917a05fd8b9b201/users/15db9d18e7b845f1838bc5f19d71b737/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:47:26.055803 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3902836-c923-4993-89a1-e0c323fb9805 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:26.056030 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3902836-c923-4993-89a1-e0c323fb9805 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:26.056320 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3902836-c923-4993-89a1-e0c323fb9805 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:26.084091 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-d3902836-c923-4993-89a1-e0c323fb9805 None None] MFA Rules not processed for user `15db9d18e7b845f1838bc5f19d71b737`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:47:26.085362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d3902836-c923-4993-89a1-e0c323fb9805 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:26.148092 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1207/2441] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:47:26 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:47:26.152683 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b14329b-6046-479c-9e49-c6724f099386 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:26.213227 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b14329b-6046-479c-9e49-c6724f099386 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:26.217835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932b63bb-c413-4916-a3ae-a63458837931 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:26.218431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932b63bb-c413-4916-a3ae-a63458837931 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:26.218711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932b63bb-c413-4916-a3ae-a63458837931 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:26.218981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932b63bb-c413-4916-a3ae-a63458837931 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:26.219617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932b63bb-c413-4916-a3ae-a63458837931 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:26.221563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932b63bb-c413-4916-a3ae-a63458837931 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:26.247816 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1235/2442] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:47:26 2022] POST /identity/v3/projects => generated 381 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:47:26.252507 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3902836-c923-4993-89a1-e0c323fb9805 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:26.263008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3902836-c923-4993-89a1-e0c323fb9805 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:26.318865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a5a14c-bca1-473a-8062-2c392a0e4eeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:26.319401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a5a14c-bca1-473a-8062-2c392a0e4eeb admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:26.319684 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a5a14c-bca1-473a-8062-2c392a0e4eeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:26.319959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a5a14c-bca1-473a-8062-2c392a0e4eeb admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:26.320630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a5a14c-bca1-473a-8062-2c392a0e4eeb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:26.322614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a5a14c-bca1-473a-8062-2c392a0e4eeb admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:26.342970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1208/2443] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:47:26 2022] POST /identity/v3/users => generated 297 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:47:26.347088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932b63bb-c413-4916-a3ae-a63458837931 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:26.357206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932b63bb-c413-4916-a3ae-a63458837931 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:26.362491 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0927f64-d12f-4e79-91b2-71f3f5efb6db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:26.363075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0927f64-d12f-4e79-91b2-71f3f5efb6db admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:26.363352 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0927f64-d12f-4e79-91b2-71f3f5efb6db admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:26.363621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0927f64-d12f-4e79-91b2-71f3f5efb6db admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:26.364256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0927f64-d12f-4e79-91b2-71f3f5efb6db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:26.365849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0927f64-d12f-4e79-91b2-71f3f5efb6db admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:26.376111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1236/2444] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:47:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:47:26.380624 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a5a14c-bca1-473a-8062-2c392a0e4eeb admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:26.390367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a5a14c-bca1-473a-8062-2c392a0e4eeb admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:26.394860 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe276fd-2765-4a6f-b392-2eba93fcc9e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:26.395496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe276fd-2765-4a6f-b392-2eba93fcc9e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:26.395770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe276fd-2765-4a6f-b392-2eba93fcc9e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:26.396041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe276fd-2765-4a6f-b392-2eba93fcc9e9 admin admin] PATH_INFO: `/v3/projects/b3781640779645e8baedd797cf05cf8a/users/ea916bb48dd14935b9b965849cfeb0e5/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:26.410339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe276fd-2765-4a6f-b392-2eba93fcc9e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3781640779645e8baedd797cf05cf8a, user_id=ea916bb48dd14935b9b965849cfeb0e5, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:26.413008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe276fd-2765-4a6f-b392-2eba93fcc9e9 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:26.427929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1209/2445] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:47:26 2022] PUT /identity/v3/projects/b3781640779645e8baedd797cf05cf8a/users/ea916bb48dd14935b9b965849cfeb0e5/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:47:26.432004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0927f64-d12f-4e79-91b2-71f3f5efb6db admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:26.442706 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0927f64-d12f-4e79-91b2-71f3f5efb6db admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:26.511007 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6977caf-ed87-4f45-b501-a787c3712ebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:26.511602 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6977caf-ed87-4f45-b501-a787c3712ebd admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:26.511877 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6977caf-ed87-4f45-b501-a787c3712ebd admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:26.512144 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6977caf-ed87-4f45-b501-a787c3712ebd admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:26.512805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6977caf-ed87-4f45-b501-a787c3712ebd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:26.514367 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6977caf-ed87-4f45-b501-a787c3712ebd admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:26.524474 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1237/2446] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:47:26 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:47:26.528665 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe276fd-2765-4a6f-b392-2eba93fcc9e9 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:26.534579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:47:26.540913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe276fd-2765-4a6f-b392-2eba93fcc9e9 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:26.545853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd092108-e1d3-45d4-9377-008bbb1d9419 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:26.546524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd092108-e1d3-45d4-9377-008bbb1d9419 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:26.546939 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd092108-e1d3-45d4-9377-008bbb1d9419 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:26.547216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd092108-e1d3-45d4-9377-008bbb1d9419 admin admin] PATH_INFO: `/v3/projects/b3781640779645e8baedd797cf05cf8a/users/ea916bb48dd14935b9b965849cfeb0e5/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:26.549857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd092108-e1d3-45d4-9377-008bbb1d9419 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3781640779645e8baedd797cf05cf8a, user_id=ea916bb48dd14935b9b965849cfeb0e5, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:26.552517 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd092108-e1d3-45d4-9377-008bbb1d9419 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:26.562496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1210/2447] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:47:26 2022] PUT /identity/v3/projects/b3781640779645e8baedd797cf05cf8a/users/ea916bb48dd14935b9b965849cfeb0e5/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:47:26.572674 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dde6142-f5a1-4b70-9a2d-7eafd3d0c4f4 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:26.572905 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dde6142-f5a1-4b70-9a2d-7eafd3d0c4f4 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:26.573178 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dde6142-f5a1-4b70-9a2d-7eafd3d0c4f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:26.600444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-8dde6142-f5a1-4b70-9a2d-7eafd3d0c4f4 None None] MFA Rules not processed for user `ea916bb48dd14935b9b965849cfeb0e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:47:26.601640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8dde6142-f5a1-4b70-9a2d-7eafd3d0c4f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:26.634379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:47:26.668672 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1238/2448] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:47:26 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:47:26.679747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd092108-e1d3-45d4-9377-008bbb1d9419 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:26.681467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-cd092108-e1d3-45d4-9377-008bbb1d9419 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:47:26.738140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd092108-e1d3-45d4-9377-008bbb1d9419 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:26.741924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7203b9f8-e175-4b66-9ba6-eba4beb761a3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:26.742546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203b9f8-e175-4b66-9ba6-eba4beb761a3 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:26.742826 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203b9f8-e175-4b66-9ba6-eba4beb761a3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:26.743100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203b9f8-e175-4b66-9ba6-eba4beb761a3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:26.751976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7203b9f8-e175-4b66-9ba6-eba4beb761a3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:26.753440 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7203b9f8-e175-4b66-9ba6-eba4beb761a3 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:26.762663 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1211/2449] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:47:26 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:47:26.891541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dde6142-f5a1-4b70-9a2d-7eafd3d0c4f4 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:47:26.901565 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dde6142-f5a1-4b70-9a2d-7eafd3d0c4f4 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:47:26.905065 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a293f9-4d83-49af-a3c2-1debfbfa050c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:47:26.905658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a293f9-4d83-49af-a3c2-1debfbfa050c service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:47:26.905935 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a293f9-4d83-49af-a3c2-1debfbfa050c service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:47:26.906207 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a293f9-4d83-49af-a3c2-1debfbfa050c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:47:26.961066 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a293f9-4d83-49af-a3c2-1debfbfa050c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:47:26.962480 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a293f9-4d83-49af-a3c2-1debfbfa050c service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:47:26.971712 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1239/2450] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:47:26 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:47:36.758539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:47:36.967141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=41 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:16.321047 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7203b9f8-e175-4b66-9ba6-eba4beb761a3 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:16.329723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7203b9f8-e175-4b66-9ba6-eba4beb761a3 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:16.333739 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9888d21-fa91-4590-b81f-4a0ed81468ab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:16.334327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9888d21-fa91-4590-b81f-4a0ed81468ab service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:16.334598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9888d21-fa91-4590-b81f-4a0ed81468ab service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:16.334864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9888d21-fa91-4590-b81f-4a0ed81468ab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:16.336640 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a9888d21-fa91-4590-b81f-4a0ed81468ab service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:16.392912 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9888d21-fa91-4590-b81f-4a0ed81468ab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:16.394365 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9888d21-fa91-4590-b81f-4a0ed81468ab service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:16.403861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1212/2451] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:16 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:17.100192 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a293f9-4d83-49af-a3c2-1debfbfa050c service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:17.108719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a293f9-4d83-49af-a3c2-1debfbfa050c service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:17.112590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6166a09b-a466-4f9f-a521-e025cbce47d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:17.113178 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6166a09b-a466-4f9f-a521-e025cbce47d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:17.113457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6166a09b-a466-4f9f-a521-e025cbce47d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:17.113729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6166a09b-a466-4f9f-a521-e025cbce47d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:17.115926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6166a09b-a466-4f9f-a521-e025cbce47d5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:17.185461 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6166a09b-a466-4f9f-a521-e025cbce47d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:17.186880 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6166a09b-a466-4f9f-a521-e025cbce47d5 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:17.198920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1240/2452] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:18.266182 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9888d21-fa91-4590-b81f-4a0ed81468ab service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:18.274713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9888d21-fa91-4590-b81f-4a0ed81468ab service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:18.278283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aeb5d9f-87f1-4b09-bc13-b8a5145642d0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:18.278866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aeb5d9f-87f1-4b09-bc13-b8a5145642d0 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:18.279142 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aeb5d9f-87f1-4b09-bc13-b8a5145642d0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:18.279438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aeb5d9f-87f1-4b09-bc13-b8a5145642d0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:18.281184 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8aeb5d9f-87f1-4b09-bc13-b8a5145642d0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:18.294457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aeb5d9f-87f1-4b09-bc13-b8a5145642d0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:18.295864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aeb5d9f-87f1-4b09-bc13-b8a5145642d0 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:18.307043 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1213/2453] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:20.517955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6166a09b-a466-4f9f-a521-e025cbce47d5 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:20.527135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6166a09b-a466-4f9f-a521-e025cbce47d5 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:20.530703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589a93a9-92ab-4e99-88c4-94fd78b487b6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:20.531321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589a93a9-92ab-4e99-88c4-94fd78b487b6 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:20.531600 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589a93a9-92ab-4e99-88c4-94fd78b487b6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:20.531872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589a93a9-92ab-4e99-88c4-94fd78b487b6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:20.533609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-589a93a9-92ab-4e99-88c4-94fd78b487b6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:20.590844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589a93a9-92ab-4e99-88c4-94fd78b487b6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:20.592258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589a93a9-92ab-4e99-88c4-94fd78b487b6 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:20.601093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1241/2454] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:20 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:28.302248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=34 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:30.597010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=62 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:35.437636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aeb5d9f-87f1-4b09-bc13-b8a5145642d0 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:35.447084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aeb5d9f-87f1-4b09-bc13-b8a5145642d0 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:35.455533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5909ef8d-2c5f-4247-b86c-64a1a9285029 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:35.456428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5909ef8d-2c5f-4247-b86c-64a1a9285029 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:35.456812 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5909ef8d-2c5f-4247-b86c-64a1a9285029 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:35.457186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5909ef8d-2c5f-4247-b86c-64a1a9285029 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:35.459751 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5909ef8d-2c5f-4247-b86c-64a1a9285029 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:35.533505 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5909ef8d-2c5f-4247-b86c-64a1a9285029 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:35.536741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5909ef8d-2c5f-4247-b86c-64a1a9285029 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:35.547677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1214/2455] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:35 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:40.475821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589a93a9-92ab-4e99-88c4-94fd78b487b6 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:40.477587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-589a93a9-92ab-4e99-88c4-94fd78b487b6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:40.555616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589a93a9-92ab-4e99-88c4-94fd78b487b6 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:40.559225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ba0756-dcb2-4894-909e-0850de7883ca service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '815504693bcc40b6aef06aacb2a9a35b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:40.559821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ba0756-dcb2-4894-909e-0850de7883ca service glance] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:40.560098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ba0756-dcb2-4894-909e-0850de7883ca service glance] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:40.560402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ba0756-dcb2-4894-909e-0850de7883ca service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:40.562098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-53ba0756-dcb2-4894-909e-0850de7883ca service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application [None req-53ba0756-dcb2-4894-909e-0850de7883ca service glance] Could not find project: 57f81fca7d7442918cb2a9e79b7d55a6.: keystone.exception.ProjectNotFound: Could not find project: 57f81fca7d7442918cb2a9e79b7d55a6. Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/auth.py", line 285, in get Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') Sep 28 12:48:40.573195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 145, in validate_token Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application token = self._validate_token(token_id) Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:48:40.574251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 179, in _validate_token Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application token.mint(token_id, issued_at) Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 579, in mint Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self._validate_token_resources() Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 464, in _validate_token_resources Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application if self.project and not self.project.get('enabled'): Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 162, in project Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application self.__project = PROVIDERS.resource_api.get_project( Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 28 12:48:40.575381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.get_or_create( Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application with Lock( Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._enter() Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.creator() Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application created_value = creator( Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 28 12:48:40.576445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 28 12:48:40.577526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: 57f81fca7d7442918cb2a9e79b7d55a6. Sep 28 12:48:40.577526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: ERROR keystone.server.flask.application Sep 28 12:48:40.577526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1242/2456] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:40 2022] GET /identity/v3/auth/tokens => generated 113 bytes in 99 msecs (HTTP/1.1 404) 5 headers in 184 bytes (1 switches on core 0) Sep 28 12:48:41.329678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5909ef8d-2c5f-4247-b86c-64a1a9285029 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:41.331121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5909ef8d-2c5f-4247-b86c-64a1a9285029 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:41.396661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5909ef8d-2c5f-4247-b86c-64a1a9285029 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:41.401452 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d4a81c-05d5-4e88-916f-4f9ba53451c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:41.402127 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d4a81c-05d5-4e88-916f-4f9ba53451c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:41.402411 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d4a81c-05d5-4e88-916f-4f9ba53451c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:41.402689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d4a81c-05d5-4e88-916f-4f9ba53451c3 admin admin] PATH_INFO: `/v3/users/0b5ecdca5a0543a1894e6462c3f2844f` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:41.403819 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d4a81c-05d5-4e88-916f-4f9ba53451c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b5ecdca5a0543a1894e6462c3f2844f)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:41.405796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d4a81c-05d5-4e88-916f-4f9ba53451c3 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:41.451697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-12d4a81c-05d5-4e88-916f-4f9ba53451c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b5ecdca5a0543a1894e6462c3f2844f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:41.452367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-12d4a81c-05d5-4e88-916f-4f9ba53451c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b5ecdca5a0543a1894e6462c3f2844f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:41.452971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-12d4a81c-05d5-4e88-916f-4f9ba53451c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b5ecdca5a0543a1894e6462c3f2844f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:41.460550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-12d4a81c-05d5-4e88-916f-4f9ba53451c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b5ecdca5a0543a1894e6462c3f2844f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:41.471913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-12d4a81c-05d5-4e88-916f-4f9ba53451c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b5ecdca5a0543a1894e6462c3f2844f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:41.485828 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1215/2457] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:41 2022] DELETE /identity/v3/users/0b5ecdca5a0543a1894e6462c3f2844f => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:41.490730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3680d-001a-408c-abe7-d4c021386125 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:41.491196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3680d-001a-408c-abe7-d4c021386125 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:41.491196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3680d-001a-408c-abe7-d4c021386125 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:41.510339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-ede3680d-001a-408c-abe7-d4c021386125 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:41.511541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ede3680d-001a-408c-abe7-d4c021386125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:41.568183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1243/2458] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:41.578429 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d4a81c-05d5-4e88-916f-4f9ba53451c3 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:41.579860 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-12d4a81c-05d5-4e88-916f-4f9ba53451c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:41.596677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:41.640862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d4a81c-05d5-4e88-916f-4f9ba53451c3 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:41.644418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7b4f10-428c-4bac-ba17-4edfae2dc8c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:41.645010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7b4f10-428c-4bac-ba17-4edfae2dc8c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:41.645871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7b4f10-428c-4bac-ba17-4edfae2dc8c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:41.646139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7b4f10-428c-4bac-ba17-4edfae2dc8c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:41.654792 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7b4f10-428c-4bac-ba17-4edfae2dc8c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:41.656212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7b4f10-428c-4bac-ba17-4edfae2dc8c3 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:41.665391 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1216/2459] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:41.935286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede3680d-001a-408c-abe7-d4c021386125 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:41.935882 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ede3680d-001a-408c-abe7-d4c021386125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:41.938200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7b4f10-428c-4bac-ba17-4edfae2dc8c3 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:41.939675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-df7b4f10-428c-4bac-ba17-4edfae2dc8c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:41.948954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede3680d-001a-408c-abe7-d4c021386125 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:41.953998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7b4f10-428c-4bac-ba17-4edfae2dc8c3 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:41.954317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de4b80c-9ae9-4e76-bad8-73183456a3ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:41.955157 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de4b80c-9ae9-4e76-bad8-73183456a3ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:41.955789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de4b80c-9ae9-4e76-bad8-73183456a3ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:41.956110 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de4b80c-9ae9-4e76-bad8-73183456a3ee admin admin] PATH_INFO: `/v3/users/9ef1cdec1a5a4c4ea1de2308d0f8de31` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:41.957186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de4b80c-9ae9-4e76-bad8-73183456a3ee admin admin] RBAC: Authorizing `identity:delete_user(user_id=9ef1cdec1a5a4c4ea1de2308d0f8de31)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:41.959215 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:41.959985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:41.960439 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:41.960545 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de4b80c-9ae9-4e76-bad8-73183456a3ee admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:41.960876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] PATH_INFO: `/v3/projects/b0eb61ddcc874275a804bfc6df1e5829` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:41.962351 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=b0eb61ddcc874275a804bfc6df1e5829)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:41.964368 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:42.020936 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4de4b80c-9ae9-4e76-bad8-73183456a3ee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9ef1cdec1a5a4c4ea1de2308d0f8de31'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.021610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4de4b80c-9ae9-4e76-bad8-73183456a3ee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9ef1cdec1a5a4c4ea1de2308d0f8de31'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.023169 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b0eb61ddcc874275a804bfc6df1e5829', 'request_id': 'req-a53858d0-c18e-4649-bdea-195087de71ea'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.028523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4de4b80c-9ae9-4e76-bad8-73183456a3ee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9ef1cdec1a5a4c4ea1de2308d0f8de31'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.029154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4de4b80c-9ae9-4e76-bad8-73183456a3ee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9ef1cdec1a5a4c4ea1de2308d0f8de31'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.029269 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b0eb61ddcc874275a804bfc6df1e5829', 'request_id': 'req-a53858d0-c18e-4649-bdea-195087de71ea'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.036511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b0eb61ddcc874275a804bfc6df1e5829', 'request_id': 'req-a53858d0-c18e-4649-bdea-195087de71ea'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.042205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4de4b80c-9ae9-4e76-bad8-73183456a3ee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9ef1cdec1a5a4c4ea1de2308d0f8de31'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.048247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] The token cache is being invalidate because project b0eb61ddcc874275a804bfc6df1e5829 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:42.048634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.049340 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.051040 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1217/2460] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:41 2022] DELETE /identity/v3/projects/b0eb61ddcc874275a804bfc6df1e5829 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:42.052395 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:42.053659 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:42.055692 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:42.061532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1244/2461] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:41 2022] DELETE /identity/v3/users/9ef1cdec1a5a4c4ea1de2308d0f8de31 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:42.065456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de4b80c-9ae9-4e76-bad8-73183456a3ee admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:42.066784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4de4b80c-9ae9-4e76-bad8-73183456a3ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:42.124926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53858d0-c18e-4649-bdea-195087de71ea admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:42.130274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb22272e-2b1f-4d14-9de2-4dc33e82da89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:42.130903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22272e-2b1f-4d14-9de2-4dc33e82da89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:42.131186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22272e-2b1f-4d14-9de2-4dc33e82da89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:42.131459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22272e-2b1f-4d14-9de2-4dc33e82da89 admin admin] PATH_INFO: `/v3/users/15db9d18e7b845f1838bc5f19d71b737` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:42.132649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb22272e-2b1f-4d14-9de2-4dc33e82da89 admin admin] RBAC: Authorizing `identity:delete_user(user_id=15db9d18e7b845f1838bc5f19d71b737)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:42.134559 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb22272e-2b1f-4d14-9de2-4dc33e82da89 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:42.135750 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de4b80c-9ae9-4e76-bad8-73183456a3ee admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:42.140272 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0500054b-e068-4b96-9f5e-e1edcc1b9818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:42.140915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500054b-e068-4b96-9f5e-e1edcc1b9818 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:42.141188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500054b-e068-4b96-9f5e-e1edcc1b9818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:42.141454 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500054b-e068-4b96-9f5e-e1edcc1b9818 admin admin] PATH_INFO: `/v3/users/5d2ee206adf54060877131b4bf12c893` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:42.142585 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0500054b-e068-4b96-9f5e-e1edcc1b9818 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5d2ee206adf54060877131b4bf12c893)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:42.144494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0500054b-e068-4b96-9f5e-e1edcc1b9818 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:42.180500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:42.187166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fb22272e-2b1f-4d14-9de2-4dc33e82da89 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '15db9d18e7b845f1838bc5f19d71b737'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.188065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fb22272e-2b1f-4d14-9de2-4dc33e82da89 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '15db9d18e7b845f1838bc5f19d71b737'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.189047 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fb22272e-2b1f-4d14-9de2-4dc33e82da89 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '15db9d18e7b845f1838bc5f19d71b737'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.195670 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0500054b-e068-4b96-9f5e-e1edcc1b9818 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5d2ee206adf54060877131b4bf12c893'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.196360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0500054b-e068-4b96-9f5e-e1edcc1b9818 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5d2ee206adf54060877131b4bf12c893'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.197572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fb22272e-2b1f-4d14-9de2-4dc33e82da89 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '15db9d18e7b845f1838bc5f19d71b737'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.203592 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0500054b-e068-4b96-9f5e-e1edcc1b9818 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5d2ee206adf54060877131b4bf12c893'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.204253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0500054b-e068-4b96-9f5e-e1edcc1b9818 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5d2ee206adf54060877131b4bf12c893'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.210836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-fb22272e-2b1f-4d14-9de2-4dc33e82da89 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '15db9d18e7b845f1838bc5f19d71b737'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.216582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0500054b-e068-4b96-9f5e-e1edcc1b9818 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5d2ee206adf54060877131b4bf12c893'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.227209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1218/2462] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:42 2022] DELETE /identity/v3/users/15db9d18e7b845f1838bc5f19d71b737 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:42.228557 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535f12d1-16c7-4ef1-a8c1-371df3609675 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:42.228825 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535f12d1-16c7-4ef1-a8c1-371df3609675 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:42.229102 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535f12d1-16c7-4ef1-a8c1-371df3609675 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:42.231971 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1245/2463] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:42 2022] DELETE /identity/v3/users/5d2ee206adf54060877131b4bf12c893 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:42.236461 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18b623a-e0fb-4b28-b9d9-b70f24361657 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:42.236748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18b623a-e0fb-4b28-b9d9-b70f24361657 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:42.237024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18b623a-e0fb-4b28-b9d9-b70f24361657 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:42.251620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-535f12d1-16c7-4ef1-a8c1-371df3609675 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:42.252904 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-535f12d1-16c7-4ef1-a8c1-371df3609675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:42.259840 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-d18b623a-e0fb-4b28-b9d9-b70f24361657 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:42.261178 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d18b623a-e0fb-4b28-b9d9-b70f24361657 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:42.334833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1219/2464] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:42.345058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535f12d1-16c7-4ef1-a8c1-371df3609675 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:42.346821 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-535f12d1-16c7-4ef1-a8c1-371df3609675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:42.350571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1246/2465] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:42.359726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18b623a-e0fb-4b28-b9d9-b70f24361657 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:42.361566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d18b623a-e0fb-4b28-b9d9-b70f24361657 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:42.386613 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:42.445354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535f12d1-16c7-4ef1-a8c1-371df3609675 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:42.449405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18b623a-e0fb-4b28-b9d9-b70f24361657 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:42.450664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1894eb61-ce7e-48da-bac4-2957ac3b4166 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:42.451613 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1894eb61-ce7e-48da-bac4-2957ac3b4166 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:42.452004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1894eb61-ce7e-48da-bac4-2957ac3b4166 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:42.452368 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1894eb61-ce7e-48da-bac4-2957ac3b4166 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:42.459461 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c843fb-72d5-41bb-a86b-0e6796fab139 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:42.460252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c843fb-72d5-41bb-a86b-0e6796fab139 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:42.460252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c843fb-72d5-41bb-a86b-0e6796fab139 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:42.462885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c843fb-72d5-41bb-a86b-0e6796fab139 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:42.464598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1894eb61-ce7e-48da-bac4-2957ac3b4166 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:42.466203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1894eb61-ce7e-48da-bac4-2957ac3b4166 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:42.473097 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c843fb-72d5-41bb-a86b-0e6796fab139 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:42.475348 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c843fb-72d5-41bb-a86b-0e6796fab139 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:42.486852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1247/2466] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:42.490216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1220/2467] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:48:42.681507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c843fb-72d5-41bb-a86b-0e6796fab139 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:42.683004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b7c843fb-72d5-41bb-a86b-0e6796fab139 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:42.695805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c843fb-72d5-41bb-a86b-0e6796fab139 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:42.700832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:42.701470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:42.701753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:42.702028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] PATH_INFO: `/v3/projects/74f642298efd4c0aa7b34eee43b5781a` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:42.703221 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=74f642298efd4c0aa7b34eee43b5781a)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:42.705236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:42.768434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74f642298efd4c0aa7b34eee43b5781a', 'request_id': 'req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.774483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74f642298efd4c0aa7b34eee43b5781a', 'request_id': 'req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.782269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74f642298efd4c0aa7b34eee43b5781a', 'request_id': 'req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.795572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] The token cache is being invalidate because project 74f642298efd4c0aa7b34eee43b5781a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:42.795863 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.796578 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.798285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1248/2468] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:42 2022] DELETE /identity/v3/projects/74f642298efd4c0aa7b34eee43b5781a => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:42.802125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1894eb61-ce7e-48da-bac4-2957ac3b4166 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:42.814819 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1894eb61-ce7e-48da-bac4-2957ac3b4166 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:42.908352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb7f585-3207-4dc6-b194-48b96acea08a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:42.908352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb7f585-3207-4dc6-b194-48b96acea08a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:42.908352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb7f585-3207-4dc6-b194-48b96acea08a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:42.908352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb7f585-3207-4dc6-b194-48b96acea08a admin admin] PATH_INFO: `/v3/users/b7dc4266ebf14ca99cad600fd37ba8df` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:42.908776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb7f585-3207-4dc6-b194-48b96acea08a admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7dc4266ebf14ca99cad600fd37ba8df)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:42.911809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb7f585-3207-4dc6-b194-48b96acea08a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:42.915201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:42.982853 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:42.984354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:42.984989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ffb7f585-3207-4dc6-b194-48b96acea08a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7dc4266ebf14ca99cad600fd37ba8df'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.986951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ffb7f585-3207-4dc6-b194-48b96acea08a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7dc4266ebf14ca99cad600fd37ba8df'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:42.988001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ffb7f585-3207-4dc6-b194-48b96acea08a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7dc4266ebf14ca99cad600fd37ba8df'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.002692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ffb7f585-3207-4dc6-b194-48b96acea08a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7dc4266ebf14ca99cad600fd37ba8df'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.010741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a2884f-1019-4ad1-b504-1d7b955ffbf3 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:43.022180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ffb7f585-3207-4dc6-b194-48b96acea08a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7dc4266ebf14ca99cad600fd37ba8df'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.031467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:43.032121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.032433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.032710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] PATH_INFO: `/v3/projects/bcc2ff8cbc234c6d9917a05fd8b9b201` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:43.033862 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] RBAC: Authorizing `identity:delete_project(project_id=bcc2ff8cbc234c6d9917a05fd8b9b201)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:43.036369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:43.043630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1221/2469] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:42 2022] DELETE /identity/v3/users/b7dc4266ebf14ca99cad600fd37ba8df => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:43.043630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb7f585-3207-4dc6-b194-48b96acea08a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:43.043630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-ffb7f585-3207-4dc6-b194-48b96acea08a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:43.091929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bcc2ff8cbc234c6d9917a05fd8b9b201', 'request_id': 'req-da03d592-1b51-4b28-8340-2b918714019f'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.101636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bcc2ff8cbc234c6d9917a05fd8b9b201', 'request_id': 'req-da03d592-1b51-4b28-8340-2b918714019f'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.110710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bcc2ff8cbc234c6d9917a05fd8b9b201', 'request_id': 'req-da03d592-1b51-4b28-8340-2b918714019f'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.122858 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb7f585-3207-4dc6-b194-48b96acea08a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:43.125250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] The token cache is being invalidate because project bcc2ff8cbc234c6d9917a05fd8b9b201 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:43.125543 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.126317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.128251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1249/2470] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:42 2022] DELETE /identity/v3/projects/bcc2ff8cbc234c6d9917a05fd8b9b201 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:43.132400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:43.133729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:43.169473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:43.194064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:43.194898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.195294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.195662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437 admin admin] PATH_INFO: `/v3/projects/557063c52e344d5383e2ceabda9bcac7` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:43.196894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437 admin admin] RBAC: Authorizing `identity:delete_project(project_id=557063c52e344d5383e2ceabda9bcac7)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:43.202267 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:43.226264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da03d592-1b51-4b28-8340-2b918714019f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:43.231079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ec58a6-acc6-47b7-b830-16971ab77c7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:43.231714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ec58a6-acc6-47b7-b830-16971ab77c7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.232047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ec58a6-acc6-47b7-b830-16971ab77c7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.232372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ec58a6-acc6-47b7-b830-16971ab77c7d admin admin] PATH_INFO: `/v3/users/ea916bb48dd14935b9b965849cfeb0e5` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:43.233673 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ec58a6-acc6-47b7-b830-16971ab77c7d admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea916bb48dd14935b9b965849cfeb0e5)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:43.235607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ec58a6-acc6-47b7-b830-16971ab77c7d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:43.246471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:43.264988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '557063c52e344d5383e2ceabda9bcac7', 'request_id': 'req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.271194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '557063c52e344d5383e2ceabda9bcac7', 'request_id': 'req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.279682 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '557063c52e344d5383e2ceabda9bcac7', 'request_id': 'req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.292873 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437 admin admin] The token cache is being invalidate because project 557063c52e344d5383e2ceabda9bcac7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:43.293217 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.293604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-24ec58a6-acc6-47b7-b830-16971ab77c7d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea916bb48dd14935b9b965849cfeb0e5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.293948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-c5c5c1ab-ad4f-411a-8e9b-31ecef2c5437 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.294263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-24ec58a6-acc6-47b7-b830-16971ab77c7d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea916bb48dd14935b9b965849cfeb0e5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.299318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1222/2471] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:43 2022] DELETE /identity/v3/projects/557063c52e344d5383e2ceabda9bcac7 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:43.300713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545bf240-852c-4db0-8c27-a98e4249eea2 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.301002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545bf240-852c-4db0-8c27-a98e4249eea2 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.301783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545bf240-852c-4db0-8c27-a98e4249eea2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:43.305861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-24ec58a6-acc6-47b7-b830-16971ab77c7d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea916bb48dd14935b9b965849cfeb0e5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.306759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-24ec58a6-acc6-47b7-b830-16971ab77c7d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea916bb48dd14935b9b965849cfeb0e5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.321705 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-24ec58a6-acc6-47b7-b830-16971ab77c7d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea916bb48dd14935b9b965849cfeb0e5'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.321906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-545bf240-852c-4db0-8c27-a98e4249eea2 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:43.323334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-545bf240-852c-4db0-8c27-a98e4249eea2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:43.338160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1250/2472] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:43 2022] DELETE /identity/v3/users/ea916bb48dd14935b9b965849cfeb0e5 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:43.362597 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ec58a6-acc6-47b7-b830-16971ab77c7d admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:43.364796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-24ec58a6-acc6-47b7-b830-16971ab77c7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:43.400287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1223/2473] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:43.405724 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906df5e0-e751-4b89-a988-957fb9a77661 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.406008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906df5e0-e751-4b89-a988-957fb9a77661 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.406267 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906df5e0-e751-4b89-a988-957fb9a77661 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:43.430328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-906df5e0-e751-4b89-a988-957fb9a77661 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:43.431539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-906df5e0-e751-4b89-a988-957fb9a77661 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:43.451970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ec58a6-acc6-47b7-b830-16971ab77c7d admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:43.456837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f72a61-b404-4393-9f6c-bebc637e8531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:43.457491 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f72a61-b404-4393-9f6c-bebc637e8531 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.458272 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f72a61-b404-4393-9f6c-bebc637e8531 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.458551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f72a61-b404-4393-9f6c-bebc637e8531 admin admin] PATH_INFO: `/v3/projects/bda425b125df4069a51181100cfc2a26` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:43.459725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f72a61-b404-4393-9f6c-bebc637e8531 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bda425b125df4069a51181100cfc2a26)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:43.461701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f72a61-b404-4393-9f6c-bebc637e8531 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:43.511995 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1224/2474] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:43.513159 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906df5e0-e751-4b89-a988-957fb9a77661 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:43.526085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906df5e0-e751-4b89-a988-957fb9a77661 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:43.532949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c8f72a61-b404-4393-9f6c-bebc637e8531 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bda425b125df4069a51181100cfc2a26', 'request_id': 'req-c8f72a61-b404-4393-9f6c-bebc637e8531'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.534471 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:43.545100 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c8f72a61-b404-4393-9f6c-bebc637e8531 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bda425b125df4069a51181100cfc2a26', 'request_id': 'req-c8f72a61-b404-4393-9f6c-bebc637e8531'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.558795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c8f72a61-b404-4393-9f6c-bebc637e8531 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bda425b125df4069a51181100cfc2a26', 'request_id': 'req-c8f72a61-b404-4393-9f6c-bebc637e8531'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.577204 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c8f72a61-b404-4393-9f6c-bebc637e8531 admin admin] The token cache is being invalidate because project bda425b125df4069a51181100cfc2a26 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:43.578112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c8f72a61-b404-4393-9f6c-bebc637e8531 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.578982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c8f72a61-b404-4393-9f6c-bebc637e8531 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.581117 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1251/2475] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:43 2022] DELETE /identity/v3/projects/bda425b125df4069a51181100cfc2a26 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:43.591838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de8b314-af85-40df-82f3-fdb03251b63d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.592223 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de8b314-af85-40df-82f3-fdb03251b63d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.593464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de8b314-af85-40df-82f3-fdb03251b63d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:43.600429 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59223b4-9225-416a-a997-1770dd6855f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:43.600966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59223b4-9225-416a-a997-1770dd6855f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.601260 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59223b4-9225-416a-a997-1770dd6855f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.601906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59223b4-9225-416a-a997-1770dd6855f6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:43.607585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59223b4-9225-416a-a997-1770dd6855f6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:48:43.607585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59223b4-9225-416a-a997-1770dd6855f6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:43.607585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59223b4-9225-416a-a997-1770dd6855f6 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:43.607585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:43.631686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-8de8b314-af85-40df-82f3-fdb03251b63d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:43.631846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1225/2476] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:48:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:48:43.631953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8de8b314-af85-40df-82f3-fdb03251b63d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:43.633134 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59223b4-9225-416a-a997-1770dd6855f6 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:43.634971 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c59223b4-9225-416a-a997-1770dd6855f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:43.646800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1252/2477] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:43.649529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9effe22-4a14-4637-8da6-254b098b773e None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.649827 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9effe22-4a14-4637-8da6-254b098b773e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.650098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9effe22-4a14-4637-8da6-254b098b773e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:43.653984 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59223b4-9225-416a-a997-1770dd6855f6 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:43.659638 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf24da9-4d20-41fc-b905-5b7676442c14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:43.660243 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf24da9-4d20-41fc-b905-5b7676442c14 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.660553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf24da9-4d20-41fc-b905-5b7676442c14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.661352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf24da9-4d20-41fc-b905-5b7676442c14 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:43.662360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf24da9-4d20-41fc-b905-5b7676442c14 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:43.664033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf24da9-4d20-41fc-b905-5b7676442c14 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:43.684494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-d9effe22-4a14-4637-8da6-254b098b773e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:43.685986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d9effe22-4a14-4637-8da6-254b098b773e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:43.696384 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1226/2478] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:43 2022] POST /identity/v3/projects => generated 403 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:43.697846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf24da9-4d20-41fc-b905-5b7676442c14 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:43.698829 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1253/2479] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:43.699799 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3bf24da9-4d20-41fc-b905-5b7676442c14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:43.700778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9effe22-4a14-4637-8da6-254b098b773e None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:43.712213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9effe22-4a14-4637-8da6-254b098b773e None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:43.779822 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67efdec1-82f4-42ad-9972-273ab9262216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:43.780576 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67efdec1-82f4-42ad-9972-273ab9262216 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.781513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67efdec1-82f4-42ad-9972-273ab9262216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.781719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67efdec1-82f4-42ad-9972-273ab9262216 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:43.782424 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67efdec1-82f4-42ad-9972-273ab9262216 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:43.784666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67efdec1-82f4-42ad-9972-273ab9262216 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:43.789979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf24da9-4d20-41fc-b905-5b7676442c14 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:43.796598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:43.797543 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.797677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.798134 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] PATH_INFO: `/v3/projects/b3781640779645e8baedd797cf05cf8a` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:43.799229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3781640779645e8baedd797cf05cf8a)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:43.801855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:43.811400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1254/2480] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:43 2022] POST /identity/v3/users => generated 302 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:48:43.812753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67efdec1-82f4-42ad-9972-273ab9262216 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:43.823252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67efdec1-82f4-42ad-9972-273ab9262216 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:43.828099 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e67f62a-a459-4f79-ae47-72ca45d4bffd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:43.828721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e67f62a-a459-4f79-ae47-72ca45d4bffd admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.829004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e67f62a-a459-4f79-ae47-72ca45d4bffd admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.829278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e67f62a-a459-4f79-ae47-72ca45d4bffd admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:43.829925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e67f62a-a459-4f79-ae47-72ca45d4bffd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:48:43.830231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e67f62a-a459-4f79-ae47-72ca45d4bffd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:43.832002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e67f62a-a459-4f79-ae47-72ca45d4bffd admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:43.846067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1255/2481] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:48:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:48:43.846892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e67f62a-a459-4f79-ae47-72ca45d4bffd admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:43.858569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e67f62a-a459-4f79-ae47-72ca45d4bffd admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:43.866752 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3781640779645e8baedd797cf05cf8a', 'request_id': 'req-44d9947f-5d43-4076-8d20-07e2d49a3f4b'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.873411 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3781640779645e8baedd797cf05cf8a', 'request_id': 'req-44d9947f-5d43-4076-8d20-07e2d49a3f4b'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.883093 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3781640779645e8baedd797cf05cf8a', 'request_id': 'req-44d9947f-5d43-4076-8d20-07e2d49a3f4b'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.898547 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] The token cache is being invalidate because project b3781640779645e8baedd797cf05cf8a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:43.898887 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.899709 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:43.902150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1227/2482] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:43 2022] DELETE /identity/v3/projects/b3781640779645e8baedd797cf05cf8a => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:43.903521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:43.905055 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:43.959448 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdef9aa-a645-402a-98bb-390b0857ff4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:43.960069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdef9aa-a645-402a-98bb-390b0857ff4e admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.960442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdef9aa-a645-402a-98bb-390b0857ff4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.960727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdef9aa-a645-402a-98bb-390b0857ff4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:43.961359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdef9aa-a645-402a-98bb-390b0857ff4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:43.963397 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdef9aa-a645-402a-98bb-390b0857ff4e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:43.976477 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1256/2483] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:43 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:43.977896 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0f422e-6c37-4550-93f2-308b5bd5cf3b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:43.978184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0f422e-6c37-4550-93f2-308b5bd5cf3b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:43.978530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0f422e-6c37-4550-93f2-308b5bd5cf3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.001084 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-0d0f422e-6c37-4550-93f2-308b5bd5cf3b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:44.002342 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0d0f422e-6c37-4550-93f2-308b5bd5cf3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:44.003055 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d9947f-5d43-4076-8d20-07e2d49a3f4b admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.009674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55630f3-5a31-4e15-90bd-44abe1b50779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.009674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55630f3-5a31-4e15-90bd-44abe1b50779 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.009674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55630f3-5a31-4e15-90bd-44abe1b50779 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.009949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55630f3-5a31-4e15-90bd-44abe1b50779 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.010363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55630f3-5a31-4e15-90bd-44abe1b50779 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.011738 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1257/2484] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:44.012275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55630f3-5a31-4e15-90bd-44abe1b50779 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.013190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0f422e-6c37-4550-93f2-308b5bd5cf3b None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.014262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0d0f422e-6c37-4550-93f2-308b5bd5cf3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:44.018837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:44.027934 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0f422e-6c37-4550-93f2-308b5bd5cf3b None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.032941 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526cc77c-86db-4a53-a479-8ebf5736307e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.033605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526cc77c-86db-4a53-a479-8ebf5736307e admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.033900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526cc77c-86db-4a53-a479-8ebf5736307e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.034249 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526cc77c-86db-4a53-a479-8ebf5736307e admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.035014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526cc77c-86db-4a53-a479-8ebf5736307e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.036835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526cc77c-86db-4a53-a479-8ebf5736307e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.043585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1228/2485] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:43 2022] POST /identity/v3/projects => generated 373 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:44.044839 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c85e71-3a89-40f4-8806-32bda55d55f5 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.045119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c85e71-3a89-40f4-8806-32bda55d55f5 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.045383 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c85e71-3a89-40f4-8806-32bda55d55f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.047893 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1258/2486] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:44 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:44.048710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526cc77c-86db-4a53-a479-8ebf5736307e admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.059258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526cc77c-86db-4a53-a479-8ebf5736307e admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.068118 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-55c85e71-3a89-40f4-8806-32bda55d55f5 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:44.069357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-55c85e71-3a89-40f4-8806-32bda55d55f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:44.076879 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:DELETE=3,keystone:SELECT=93,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:44.119795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc438ac-8287-4dcf-8a06-4071b8bdd889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.120425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc438ac-8287-4dcf-8a06-4071b8bdd889 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.120704 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc438ac-8287-4dcf-8a06-4071b8bdd889 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.120967 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc438ac-8287-4dcf-8a06-4071b8bdd889 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.121589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc438ac-8287-4dcf-8a06-4071b8bdd889 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.123482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc438ac-8287-4dcf-8a06-4071b8bdd889 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.130930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1229/2487] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:44.131752 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c85e71-3a89-40f4-8806-32bda55d55f5 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.141715 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c85e71-3a89-40f4-8806-32bda55d55f5 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.149298 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1259/2488] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:44 2022] POST /identity/v3/users => generated 287 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:44.149445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33d8396-3bd1-49a0-b063-5dad6a777839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.149540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc438ac-8287-4dcf-8a06-4071b8bdd889 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.149622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33d8396-3bd1-49a0-b063-5dad6a777839 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.149622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33d8396-3bd1-49a0-b063-5dad6a777839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.149622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33d8396-3bd1-49a0-b063-5dad6a777839 admin admin] PATH_INFO: `/v3/projects/140b03e3006e4633b718455b69e1edd6/users/a2fd45ad5b0146e3a38df4963fe31520/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.156691 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc438ac-8287-4dcf-8a06-4071b8bdd889 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.161998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bcc508-29a7-456e-9214-1cf56c2153fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.162306 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33d8396-3bd1-49a0-b063-5dad6a777839 admin admin] RBAC: Authorizing `identity:create_grant(project_id=140b03e3006e4633b718455b69e1edd6, user_id=a2fd45ad5b0146e3a38df4963fe31520, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.162581 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bcc508-29a7-456e-9214-1cf56c2153fd admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.162861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bcc508-29a7-456e-9214-1cf56c2153fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.163133 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bcc508-29a7-456e-9214-1cf56c2153fd admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.163782 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bcc508-29a7-456e-9214-1cf56c2153fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:48:44.164084 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bcc508-29a7-456e-9214-1cf56c2153fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.165076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33d8396-3bd1-49a0-b063-5dad6a777839 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.165691 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bcc508-29a7-456e-9214-1cf56c2153fd admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.181157 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1260/2489] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:48:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:48:44.181781 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bcc508-29a7-456e-9214-1cf56c2153fd admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.182102 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1230/2490] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:44 2022] PUT /identity/v3/projects/140b03e3006e4633b718455b69e1edd6/users/a2fd45ad5b0146e3a38df4963fe31520/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:44.187983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded80eea-b9cb-42e2-b8ed-4ad56f28430b None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.188153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded80eea-b9cb-42e2-b8ed-4ad56f28430b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.188470 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded80eea-b9cb-42e2-b8ed-4ad56f28430b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.196539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bcc508-29a7-456e-9214-1cf56c2153fd admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.208663 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-ded80eea-b9cb-42e2-b8ed-4ad56f28430b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:44.209853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-ded80eea-b9cb-42e2-b8ed-4ad56f28430b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:44.254495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c04e4fd-657c-4364-b0b1-3933c50ba4cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.255596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c04e4fd-657c-4364-b0b1-3933c50ba4cc admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.255900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c04e4fd-657c-4364-b0b1-3933c50ba4cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.256170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c04e4fd-657c-4364-b0b1-3933c50ba4cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.256945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c04e4fd-657c-4364-b0b1-3933c50ba4cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.258518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c04e4fd-657c-4364-b0b1-3933c50ba4cc admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.271049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1261/2491] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:44 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:44.271049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c04e4fd-657c-4364-b0b1-3933c50ba4cc admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.273034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1231/2492] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:48:44.273979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded80eea-b9cb-42e2-b8ed-4ad56f28430b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.278903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c04e4fd-657c-4364-b0b1-3933c50ba4cc admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.283488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded80eea-b9cb-42e2-b8ed-4ad56f28430b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.283771 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ec3688-63b3-4004-93d7-e9f4d1ec3d78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.284382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ec3688-63b3-4004-93d7-e9f4d1ec3d78 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.284658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ec3688-63b3-4004-93d7-e9f4d1ec3d78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.284925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ec3688-63b3-4004-93d7-e9f4d1ec3d78 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.285546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ec3688-63b3-4004-93d7-e9f4d1ec3d78 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.287478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ec3688-63b3-4004-93d7-e9f4d1ec3d78 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.288233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712332b8-ff8b-41a9-9943-5cd3e3735c5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.288854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712332b8-ff8b-41a9-9943-5cd3e3735c5a admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.289131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712332b8-ff8b-41a9-9943-5cd3e3735c5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.289402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712332b8-ff8b-41a9-9943-5cd3e3735c5a admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.290045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712332b8-ff8b-41a9-9943-5cd3e3735c5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.291622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712332b8-ff8b-41a9-9943-5cd3e3735c5a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.302085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1232/2493] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:44 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:44.302879 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712332b8-ff8b-41a9-9943-5cd3e3735c5a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.306694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-712332b8-ff8b-41a9-9943-5cd3e3735c5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:44.316592 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1262/2494] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:44 2022] POST /identity/v3/projects => generated 367 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:44.318083 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ec3688-63b3-4004-93d7-e9f4d1ec3d78 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.368554 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712332b8-ff8b-41a9-9943-5cd3e3735c5a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.373136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:44.380961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ec3688-63b3-4004-93d7-e9f4d1ec3d78 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.385784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74677b4c-5bf2-4ed4-b4eb-80409848f9f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.386427 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74677b4c-5bf2-4ed4-b4eb-80409848f9f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.386703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74677b4c-5bf2-4ed4-b4eb-80409848f9f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.386978 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74677b4c-5bf2-4ed4-b4eb-80409848f9f5 admin admin] PATH_INFO: `/v3/projects/5b99b160db0f4b24a875642de028f2f8/users/0d9d9929db5f4150a12b29b0b39a8eaf/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.402076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74677b4c-5bf2-4ed4-b4eb-80409848f9f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b99b160db0f4b24a875642de028f2f8, user_id=0d9d9929db5f4150a12b29b0b39a8eaf, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.404995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74677b4c-5bf2-4ed4-b4eb-80409848f9f5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.407769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:44.423289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1263/2495] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:44 2022] PUT /identity/v3/projects/5b99b160db0f4b24a875642de028f2f8/users/0d9d9929db5f4150a12b29b0b39a8eaf/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:44.424068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74677b4c-5bf2-4ed4-b4eb-80409848f9f5 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.425206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b682ca5f-356c-45e6-9cd6-9ce65ad514ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.425822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b682ca5f-356c-45e6-9cd6-9ce65ad514ab service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.426103 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b682ca5f-356c-45e6-9cd6-9ce65ad514ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.426371 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b682ca5f-356c-45e6-9cd6-9ce65ad514ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.436603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b682ca5f-356c-45e6-9cd6-9ce65ad514ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.438042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b682ca5f-356c-45e6-9cd6-9ce65ad514ab service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.448755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1233/2496] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:44.450019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d187ce96-c4a3-48e9-9f59-8443a9531d21 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.450306 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d187ce96-c4a3-48e9-9f59-8443a9531d21 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.450580 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d187ce96-c4a3-48e9-9f59-8443a9531d21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.474692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-d187ce96-c4a3-48e9-9f59-8443a9531d21 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:44.476459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d187ce96-c4a3-48e9-9f59-8443a9531d21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:44.500032 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74677b4c-5bf2-4ed4-b4eb-80409848f9f5 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.506381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abd3379-f61f-40b8-a856-446c50da1da0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.506990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abd3379-f61f-40b8-a856-446c50da1da0 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.507274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abd3379-f61f-40b8-a856-446c50da1da0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.507558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abd3379-f61f-40b8-a856-446c50da1da0 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.508507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abd3379-f61f-40b8-a856-446c50da1da0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.510542 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abd3379-f61f-40b8-a856-446c50da1da0 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.536403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1264/2497] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:44 2022] POST /identity/v3/users => generated 284 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:44.540633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abd3379-f61f-40b8-a856-446c50da1da0 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.552940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abd3379-f61f-40b8-a856-446c50da1da0 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.555539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1234/2498] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:44.558695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0887e359-312d-4b37-a705-3fed86f8d217 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.559481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0887e359-312d-4b37-a705-3fed86f8d217 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.559563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0887e359-312d-4b37-a705-3fed86f8d217 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.559836 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0887e359-312d-4b37-a705-3fed86f8d217 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.560550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0887e359-312d-4b37-a705-3fed86f8d217 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.562147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0887e359-312d-4b37-a705-3fed86f8d217 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.565799 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d187ce96-c4a3-48e9-9f59-8443a9531d21 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.568211 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d187ce96-c4a3-48e9-9f59-8443a9531d21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:44.575314 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1265/2499] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:44 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:44.579151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0887e359-312d-4b37-a705-3fed86f8d217 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.591725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0887e359-312d-4b37-a705-3fed86f8d217 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.598934 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c45b51-0b7f-4390-8364-58bceb4558a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.599541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c45b51-0b7f-4390-8364-58bceb4558a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.599830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c45b51-0b7f-4390-8364-58bceb4558a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.600107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c45b51-0b7f-4390-8364-58bceb4558a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.601131 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c45b51-0b7f-4390-8364-58bceb4558a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.603594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c45b51-0b7f-4390-8364-58bceb4558a3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.624665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1266/2500] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:44 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:48:44.625865 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c45b51-0b7f-4390-8364-58bceb4558a3 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.638541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c45b51-0b7f-4390-8364-58bceb4558a3 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.643194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0260af9-ce09-466b-806d-6dad04019c30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.644060 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0260af9-ce09-466b-806d-6dad04019c30 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.644365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0260af9-ce09-466b-806d-6dad04019c30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.644642 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0260af9-ce09-466b-806d-6dad04019c30 admin admin] PATH_INFO: `/v3/projects/56ed67dbbfdd4159925f9a25b37d52f4/users/0430169e743e4d00a49c5d24b0d48df7/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.645106 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d187ce96-c4a3-48e9-9f59-8443a9531d21 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.650540 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f92109-7aea-4c6b-8a42-37acf4314f30 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.651155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f92109-7aea-4c6b-8a42-37acf4314f30 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.651441 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f92109-7aea-4c6b-8a42-37acf4314f30 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.651881 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f92109-7aea-4c6b-8a42-37acf4314f30 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.662374 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0260af9-ce09-466b-806d-6dad04019c30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56ed67dbbfdd4159925f9a25b37d52f4, user_id=0430169e743e4d00a49c5d24b0d48df7, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.663100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f92109-7aea-4c6b-8a42-37acf4314f30 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.664830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f92109-7aea-4c6b-8a42-37acf4314f30 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.665770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0260af9-ce09-466b-806d-6dad04019c30 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:44.676732 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1235/2501] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:48:44.681276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1267/2502] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:44 2022] PUT /identity/v3/projects/56ed67dbbfdd4159925f9a25b37d52f4/users/0430169e743e4d00a49c5d24b0d48df7/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:44.693034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb76b99-ebbd-499d-ad1f-9dedcb40a372 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.693283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb76b99-ebbd-499d-ad1f-9dedcb40a372 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.693556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb76b99-ebbd-499d-ad1f-9dedcb40a372 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.730355 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-bdb76b99-ebbd-499d-ad1f-9dedcb40a372 None None] MFA Rules not processed for user `0430169e743e4d00a49c5d24b0d48df7`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:44.731641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-bdb76b99-ebbd-499d-ad1f-9dedcb40a372 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:44.804350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:44.847035 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1236/2503] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:44 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:44.852448 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0260af9-ce09-466b-806d-6dad04019c30 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:44.943226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0260af9-ce09-466b-806d-6dad04019c30 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:44.949567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff22cc68-ab06-4e88-adfd-ceb6186cc7c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:44.950217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff22cc68-ab06-4e88-adfd-ceb6186cc7c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:44.950505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff22cc68-ab06-4e88-adfd-ceb6186cc7c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:44.950777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff22cc68-ab06-4e88-adfd-ceb6186cc7c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:44.951717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff22cc68-ab06-4e88-adfd-ceb6186cc7c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:44.953682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff22cc68-ab06-4e88-adfd-ceb6186cc7c3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:45.040794 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1268/2504] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:44 2022] POST /identity/v3/projects => generated 367 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:45.044616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb76b99-ebbd-499d-ad1f-9dedcb40a372 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:45.059497 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb76b99-ebbd-499d-ad1f-9dedcb40a372 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:45.174381 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa14505d-ca35-4481-a706-b6fce44199aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:45.174986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa14505d-ca35-4481-a706-b6fce44199aa admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:45.175264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa14505d-ca35-4481-a706-b6fce44199aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:45.175534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa14505d-ca35-4481-a706-b6fce44199aa admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:45.176179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa14505d-ca35-4481-a706-b6fce44199aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:45.178677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa14505d-ca35-4481-a706-b6fce44199aa admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:45.205108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1237/2505] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:45 2022] POST /identity/v3/users => generated 290 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:48:45.209776 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff22cc68-ab06-4e88-adfd-ceb6186cc7c3 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:45.220910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff22cc68-ab06-4e88-adfd-ceb6186cc7c3 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:45.233414 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec4cff6-da33-40a3-b9a9-b7f31cb3fa25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:45.234215 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec4cff6-da33-40a3-b9a9-b7f31cb3fa25 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:45.234591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec4cff6-da33-40a3-b9a9-b7f31cb3fa25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:45.234888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec4cff6-da33-40a3-b9a9-b7f31cb3fa25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:45.236446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec4cff6-da33-40a3-b9a9-b7f31cb3fa25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:45.238287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec4cff6-da33-40a3-b9a9-b7f31cb3fa25 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:45.255825 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1269/2506] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:45 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:45.257064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa14505d-ca35-4481-a706-b6fce44199aa admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:45.272108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa14505d-ca35-4481-a706-b6fce44199aa admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:45.277769 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1004448c-46e4-47a5-ae76-982245055d52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:45.278436 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1004448c-46e4-47a5-ae76-982245055d52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:45.278716 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1004448c-46e4-47a5-ae76-982245055d52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:45.278994 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1004448c-46e4-47a5-ae76-982245055d52 admin admin] PATH_INFO: `/v3/projects/7aa98378563f40fda40166fd622af528/users/880fc2d17e5c405daf719c589234413a/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:45.297842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1004448c-46e4-47a5-ae76-982245055d52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7aa98378563f40fda40166fd622af528, user_id=880fc2d17e5c405daf719c589234413a, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:45.300451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1004448c-46e4-47a5-ae76-982245055d52 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:45.316693 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1238/2507] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:45 2022] PUT /identity/v3/projects/7aa98378563f40fda40166fd622af528/users/880fc2d17e5c405daf719c589234413a/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:45.320656 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec4cff6-da33-40a3-b9a9-b7f31cb3fa25 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:45.330916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec4cff6-da33-40a3-b9a9-b7f31cb3fa25 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:45.339445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:45.403951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74337f9b-dedb-4f55-bfec-e4ff87a24c6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:45.404616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74337f9b-dedb-4f55-bfec-e4ff87a24c6b admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:45.404900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74337f9b-dedb-4f55-bfec-e4ff87a24c6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:45.405179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74337f9b-dedb-4f55-bfec-e4ff87a24c6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:45.405847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74337f9b-dedb-4f55-bfec-e4ff87a24c6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:45.407520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74337f9b-dedb-4f55-bfec-e4ff87a24c6b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:45.417238 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb42d87-0bcb-41ed-80b7-a9c8c2a1affa None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:45.417546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb42d87-0bcb-41ed-80b7-a9c8c2a1affa None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:45.417798 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb42d87-0bcb-41ed-80b7-a9c8c2a1affa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:45.432390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1270/2508] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:45 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:48:45.435388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74337f9b-dedb-4f55-bfec-e4ff87a24c6b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:45.449129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74337f9b-dedb-4f55-bfec-e4ff87a24c6b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:45.449923 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-ffb42d87-0bcb-41ed-80b7-a9c8c2a1affa None None] MFA Rules not processed for user `a2fd45ad5b0146e3a38df4963fe31520`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:45.451296 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-ffb42d87-0bcb-41ed-80b7-a9c8c2a1affa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:45.455586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e15a11-8355-453b-82cf-2e5427a9f22d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:45.457141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e15a11-8355-453b-82cf-2e5427a9f22d admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:45.458352 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e15a11-8355-453b-82cf-2e5427a9f22d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:45.458938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e15a11-8355-453b-82cf-2e5427a9f22d admin admin] PATH_INFO: `/v3/projects/7aa98378563f40fda40166fd622af528/users/880fc2d17e5c405daf719c589234413a/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:45.461347 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e15a11-8355-453b-82cf-2e5427a9f22d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7aa98378563f40fda40166fd622af528, user_id=880fc2d17e5c405daf719c589234413a, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:45.464273 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e15a11-8355-453b-82cf-2e5427a9f22d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:45.492695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1271/2509] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:45 2022] PUT /identity/v3/projects/7aa98378563f40fda40166fd622af528/users/880fc2d17e5c405daf719c589234413a/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:45.496090 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7f11c3-1ac2-4dfa-baf2-22fc4b0ec70a None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:45.496428 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7f11c3-1ac2-4dfa-baf2-22fc4b0ec70a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:45.496634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7f11c3-1ac2-4dfa-baf2-22fc4b0ec70a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:45.538626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-aa7f11c3-1ac2-4dfa-baf2-22fc4b0ec70a None None] MFA Rules not processed for user `880fc2d17e5c405daf719c589234413a`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:45.541427 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-aa7f11c3-1ac2-4dfa-baf2-22fc4b0ec70a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:45.548743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1239/2510] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:45 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:45.553343 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb42d87-0bcb-41ed-80b7-a9c8c2a1affa None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:45.565970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb42d87-0bcb-41ed-80b7-a9c8c2a1affa None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:45.624269 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:45.642947 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1272/2511] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:45 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:45.644214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdf1e34-1b0d-4871-998d-fabc2eea41f5 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:45.644544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdf1e34-1b0d-4871-998d-fabc2eea41f5 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:45.644820 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdf1e34-1b0d-4871-998d-fabc2eea41f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:45.650341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9882c52-bb5f-47dd-aea9-5468470e20c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:45.651093 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9882c52-bb5f-47dd-aea9-5468470e20c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:45.651426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9882c52-bb5f-47dd-aea9-5468470e20c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:45.651762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9882c52-bb5f-47dd-aea9-5468470e20c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:45.652501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9882c52-bb5f-47dd-aea9-5468470e20c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:45.654484 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9882c52-bb5f-47dd-aea9-5468470e20c4 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:45.682007 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-9fdf1e34-1b0d-4871-998d-fabc2eea41f5 None None] MFA Rules not processed for user `0d9d9929db5f4150a12b29b0b39a8eaf`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:45.683458 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-9fdf1e34-1b0d-4871-998d-fabc2eea41f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:45.684760 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1240/2512] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:45 2022] POST /identity/v3/projects => generated 403 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:45.685679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9882c52-bb5f-47dd-aea9-5468470e20c4 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:45.689270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e9882c52-bb5f-47dd-aea9-5468470e20c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:45.758944 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9882c52-bb5f-47dd-aea9-5468470e20c4 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:45.762805 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f27c415-6abb-45cf-b38e-e9105067521f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:45.763513 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f27c415-6abb-45cf-b38e-e9105067521f service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:45.763863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f27c415-6abb-45cf-b38e-e9105067521f service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:45.764230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f27c415-6abb-45cf-b38e-e9105067521f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:45.777416 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1273/2513] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:45 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:45.777416 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdf1e34-1b0d-4871-998d-fabc2eea41f5 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:45.788650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:45.840557 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f27c415-6abb-45cf-b38e-e9105067521f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:45.841916 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f27c415-6abb-45cf-b38e-e9105067521f service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:45.847470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdf1e34-1b0d-4871-998d-fabc2eea41f5 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:45.853171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1241/2514] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:45 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:45.853587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d47c9b-8d14-468c-8408-edb1fc2f3183 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:45.854201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d47c9b-8d14-468c-8408-edb1fc2f3183 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:45.854676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d47c9b-8d14-468c-8408-edb1fc2f3183 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:45.854963 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d47c9b-8d14-468c-8408-edb1fc2f3183 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:45.855601 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d47c9b-8d14-468c-8408-edb1fc2f3183 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:45.856525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f27c415-6abb-45cf-b38e-e9105067521f service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:45.857577 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d47c9b-8d14-468c-8408-edb1fc2f3183 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:45.871455 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f27c415-6abb-45cf-b38e-e9105067521f service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:45.877769 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9812c3-664d-4dd0-aa67-abc4e75ec432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:45.878412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9812c3-664d-4dd0-aa67-abc4e75ec432 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:45.879122 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9812c3-664d-4dd0-aa67-abc4e75ec432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:45.880015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9812c3-664d-4dd0-aa67-abc4e75ec432 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:45.880885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9812c3-664d-4dd0-aa67-abc4e75ec432 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:45.883191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9812c3-664d-4dd0-aa67-abc4e75ec432 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:45.891724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1274/2515] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:45 2022] POST /identity/v3/users => generated 308 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:45.895463 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d47c9b-8d14-468c-8408-edb1fc2f3183 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:45.907255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d47c9b-8d14-468c-8408-edb1fc2f3183 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:45.919826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2818688-d7ec-4ea2-afa6-7ed9a91ff255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:45.921115 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2818688-d7ec-4ea2-afa6-7ed9a91ff255 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:45.921513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2818688-d7ec-4ea2-afa6-7ed9a91ff255 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:45.922342 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2818688-d7ec-4ea2-afa6-7ed9a91ff255 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:45.922570 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2818688-d7ec-4ea2-afa6-7ed9a91ff255 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:45.924456 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2818688-d7ec-4ea2-afa6-7ed9a91ff255 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:45.937609 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1242/2516] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:45 2022] POST /identity/v3/projects => generated 373 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:45.941381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1275/2517] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:45 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:45.943017 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9812c3-664d-4dd0-aa67-abc4e75ec432 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:45.943321 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2818688-d7ec-4ea2-afa6-7ed9a91ff255 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:45.962650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2818688-d7ec-4ea2-afa6-7ed9a91ff255 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:46.007754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:46.020030 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9812c3-664d-4dd0-aa67-abc4e75ec432 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:46.025742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bca68e8-10fd-4876-b56e-0e3ff48cc1b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:46.026087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862eaf9d-f24c-4f59-8bbb-fbfd87a4d0bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:46.026349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bca68e8-10fd-4876-b56e-0e3ff48cc1b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:46.026630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bca68e8-10fd-4876-b56e-0e3ff48cc1b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:46.026741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862eaf9d-f24c-4f59-8bbb-fbfd87a4d0bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:46.026904 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bca68e8-10fd-4876-b56e-0e3ff48cc1b7 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:46.027040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862eaf9d-f24c-4f59-8bbb-fbfd87a4d0bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:46.027322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862eaf9d-f24c-4f59-8bbb-fbfd87a4d0bf admin admin] PATH_INFO: `/v3/projects/40521248b9c94502aaa5ca21d56ec6db/users/c7b5a974044648e4ab5f7706f1622b32/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:46.027534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bca68e8-10fd-4876-b56e-0e3ff48cc1b7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:46.029515 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bca68e8-10fd-4876-b56e-0e3ff48cc1b7 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:46.043416 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862eaf9d-f24c-4f59-8bbb-fbfd87a4d0bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=40521248b9c94502aaa5ca21d56ec6db, user_id=c7b5a974044648e4ab5f7706f1622b32, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:46.046512 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862eaf9d-f24c-4f59-8bbb-fbfd87a4d0bf admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:46.054399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1243/2518] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:45 2022] POST /identity/v3/users => generated 293 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:46.055361 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bca68e8-10fd-4876-b56e-0e3ff48cc1b7 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:46.065047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1276/2519] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:45 2022] PUT /identity/v3/projects/40521248b9c94502aaa5ca21d56ec6db/users/c7b5a974044648e4ab5f7706f1622b32/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:46.066277 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862eaf9d-f24c-4f59-8bbb-fbfd87a4d0bf admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:46.117530 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bca68e8-10fd-4876-b56e-0e3ff48cc1b7 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:46.137681 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862eaf9d-f24c-4f59-8bbb-fbfd87a4d0bf admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:46.142799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226f0eaf-b1d3-4ec3-ad52-2a951e0e19c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:46.143407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226f0eaf-b1d3-4ec3-ad52-2a951e0e19c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:46.143692 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226f0eaf-b1d3-4ec3-ad52-2a951e0e19c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:46.143966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226f0eaf-b1d3-4ec3-ad52-2a951e0e19c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:46.144647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226f0eaf-b1d3-4ec3-ad52-2a951e0e19c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:46.146295 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226f0eaf-b1d3-4ec3-ad52-2a951e0e19c3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:46.156567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1277/2520] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:46 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:46.157362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226f0eaf-b1d3-4ec3-ad52-2a951e0e19c3 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:46.170318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226f0eaf-b1d3-4ec3-ad52-2a951e0e19c3 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:46.174972 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac3a889-ff23-477e-a55f-b6fb85497c6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:46.175558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac3a889-ff23-477e-a55f-b6fb85497c6d admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:46.175843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac3a889-ff23-477e-a55f-b6fb85497c6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:46.176121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac3a889-ff23-477e-a55f-b6fb85497c6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:46.176955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2b8dc6-60e9-4ac9-a32c-8f46c57be953 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:46.178398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2b8dc6-60e9-4ac9-a32c-8f46c57be953 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:46.179130 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac3a889-ff23-477e-a55f-b6fb85497c6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:46.179219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2b8dc6-60e9-4ac9-a32c-8f46c57be953 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:46.179219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2b8dc6-60e9-4ac9-a32c-8f46c57be953 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:46.180539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac3a889-ff23-477e-a55f-b6fb85497c6d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:46.182529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:46.196523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1278/2521] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:46 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:46.196523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac3a889-ff23-477e-a55f-b6fb85497c6d admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:46.204640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac3a889-ff23-477e-a55f-b6fb85497c6d admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:46.209836 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb7ffba-ba02-44dd-9429-0c53fd67ea8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:46.210472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb7ffba-ba02-44dd-9429-0c53fd67ea8f admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:46.210751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb7ffba-ba02-44dd-9429-0c53fd67ea8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:46.211027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb7ffba-ba02-44dd-9429-0c53fd67ea8f admin admin] PATH_INFO: `/v3/projects/ae9d113dac8742fc9e957c2b7f6ce7c5/users/18d2841c2aa241abbaad07612ce5ea03/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:46.225694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb7ffba-ba02-44dd-9429-0c53fd67ea8f admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae9d113dac8742fc9e957c2b7f6ce7c5, user_id=18d2841c2aa241abbaad07612ce5ea03, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:46.228362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb7ffba-ba02-44dd-9429-0c53fd67ea8f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:46.237316 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2b8dc6-60e9-4ac9-a32c-8f46c57be953 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:46.238607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2b8dc6-60e9-4ac9-a32c-8f46c57be953 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:46.242196 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1279/2522] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:46 2022] PUT /identity/v3/projects/ae9d113dac8742fc9e957c2b7f6ce7c5/users/18d2841c2aa241abbaad07612ce5ea03/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:46.242986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb7ffba-ba02-44dd-9429-0c53fd67ea8f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:46.248677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1244/2523] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:46 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:46.249707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2b8dc6-60e9-4ac9-a32c-8f46c57be953 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:46.259836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2b8dc6-60e9-4ac9-a32c-8f46c57be953 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:46.316443 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb7ffba-ba02-44dd-9429-0c53fd67ea8f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:46.323422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563f921b-36d5-49b0-91fa-ce107ec212eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:46.324191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563f921b-36d5-49b0-91fa-ce107ec212eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:46.324864 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563f921b-36d5-49b0-91fa-ce107ec212eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:46.325184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563f921b-36d5-49b0-91fa-ce107ec212eb admin admin] PATH_INFO: `/v3/projects/40521248b9c94502aaa5ca21d56ec6db/users/c7b5a974044648e4ab5f7706f1622b32/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:46.327327 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563f921b-36d5-49b0-91fa-ce107ec212eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=40521248b9c94502aaa5ca21d56ec6db, user_id=c7b5a974044648e4ab5f7706f1622b32, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:46.328174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:46.330595 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563f921b-36d5-49b0-91fa-ce107ec212eb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:46.341434 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e148a935-58f6-495a-8746-3fc57e5f1252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:46.342046 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e148a935-58f6-495a-8746-3fc57e5f1252 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:46.342323 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e148a935-58f6-495a-8746-3fc57e5f1252 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:46.342621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e148a935-58f6-495a-8746-3fc57e5f1252 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:46.346607 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1280/2524] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:46 2022] PUT /identity/v3/projects/40521248b9c94502aaa5ca21d56ec6db/users/c7b5a974044648e4ab5f7706f1622b32/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:46.346758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e148a935-58f6-495a-8746-3fc57e5f1252 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:46.346758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e148a935-58f6-495a-8746-3fc57e5f1252 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:46.358498 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1245/2525] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:46 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:46.363539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563f921b-36d5-49b0-91fa-ce107ec212eb admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:46.435690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563f921b-36d5-49b0-91fa-ce107ec212eb admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:46.441417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aadce99-ba5b-47e6-ba85-4e095dd4e79c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:46.442048 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aadce99-ba5b-47e6-ba85-4e095dd4e79c admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:46.442610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aadce99-ba5b-47e6-ba85-4e095dd4e79c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:46.442740 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aadce99-ba5b-47e6-ba85-4e095dd4e79c admin admin] PATH_INFO: `/v3/projects/ae9d113dac8742fc9e957c2b7f6ce7c5/users/18d2841c2aa241abbaad07612ce5ea03/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:46.445056 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aadce99-ba5b-47e6-ba85-4e095dd4e79c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae9d113dac8742fc9e957c2b7f6ce7c5, user_id=18d2841c2aa241abbaad07612ce5ea03, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:46.447945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aadce99-ba5b-47e6-ba85-4e095dd4e79c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:46.465974 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1281/2526] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:46 2022] PUT /identity/v3/projects/ae9d113dac8742fc9e957c2b7f6ce7c5/users/18d2841c2aa241abbaad07612ce5ea03/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:47.101466 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d9eca2-484b-4031-8bf8-2720c6bf7fcb None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:47.102075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d9eca2-484b-4031-8bf8-2720c6bf7fcb None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:47.102075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d9eca2-484b-4031-8bf8-2720c6bf7fcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:47.155200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-67d9eca2-484b-4031-8bf8-2720c6bf7fcb None None] MFA Rules not processed for user `c7b5a974044648e4ab5f7706f1622b32`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:47.157119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-67d9eca2-484b-4031-8bf8-2720c6bf7fcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:47.255654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1246/2527] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:47 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:47.416804 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec0cbed-27a3-4953-bb98-0d4dc691d2b7 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:47.417312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec0cbed-27a3-4953-bb98-0d4dc691d2b7 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:47.417911 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec0cbed-27a3-4953-bb98-0d4dc691d2b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:47.448720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-dec0cbed-27a3-4953-bb98-0d4dc691d2b7 None None] MFA Rules not processed for user `18d2841c2aa241abbaad07612ce5ea03`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:47.449950 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-dec0cbed-27a3-4953-bb98-0d4dc691d2b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:47.511035 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:47.542497 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1282/2528] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:47 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:47.555083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d9eca2-484b-4031-8bf8-2720c6bf7fcb None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:47.611423 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d9eca2-484b-4031-8bf8-2720c6bf7fcb None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:47.615181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11406ecb-9b69-4360-9c7c-fa4a08a9d4f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:47.615841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11406ecb-9b69-4360-9c7c-fa4a08a9d4f7 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:47.616083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11406ecb-9b69-4360-9c7c-fa4a08a9d4f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:47.616340 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11406ecb-9b69-4360-9c7c-fa4a08a9d4f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:47.682250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11406ecb-9b69-4360-9c7c-fa4a08a9d4f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:47.683619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11406ecb-9b69-4360-9c7c-fa4a08a9d4f7 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:47.693356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1247/2529] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:47 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:48.042979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec0cbed-27a3-4953-bb98-0d4dc691d2b7 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:48.044407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-dec0cbed-27a3-4953-bb98-0d4dc691d2b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:48.133081 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec0cbed-27a3-4953-bb98-0d4dc691d2b7 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:48.138153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702605b7-d517-4f0a-9d63-b90298e79717 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:48.138762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702605b7-d517-4f0a-9d63-b90298e79717 service placement] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:48.139043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702605b7-d517-4f0a-9d63-b90298e79717 service placement] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:48.139318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702605b7-d517-4f0a-9d63-b90298e79717 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:48.141054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-702605b7-d517-4f0a-9d63-b90298e79717 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:48.152941 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702605b7-d517-4f0a-9d63-b90298e79717 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:48.154736 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702605b7-d517-4f0a-9d63-b90298e79717 service placement] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:48.165303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1283/2530] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:48 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:48.612425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11406ecb-9b69-4360-9c7c-fa4a08a9d4f7 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:48.646613 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:48.678899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11406ecb-9b69-4360-9c7c-fa4a08a9d4f7 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:48.683531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363bc380-5ba2-4472-bdbe-a54567fcdbc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:48.684148 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363bc380-5ba2-4472-bdbe-a54567fcdbc3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:48.684459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363bc380-5ba2-4472-bdbe-a54567fcdbc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:48.684738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363bc380-5ba2-4472-bdbe-a54567fcdbc3 admin admin] PATH_INFO: `/v3/users/a2fd45ad5b0146e3a38df4963fe31520` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:48.685956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363bc380-5ba2-4472-bdbe-a54567fcdbc3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2fd45ad5b0146e3a38df4963fe31520)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:48.687931 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363bc380-5ba2-4472-bdbe-a54567fcdbc3 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:48.744724 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-363bc380-5ba2-4472-bdbe-a54567fcdbc3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2fd45ad5b0146e3a38df4963fe31520'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:48.745294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-363bc380-5ba2-4472-bdbe-a54567fcdbc3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2fd45ad5b0146e3a38df4963fe31520'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:48.745887 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-363bc380-5ba2-4472-bdbe-a54567fcdbc3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2fd45ad5b0146e3a38df4963fe31520'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:48.752633 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-363bc380-5ba2-4472-bdbe-a54567fcdbc3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2fd45ad5b0146e3a38df4963fe31520'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:48.764733 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-363bc380-5ba2-4472-bdbe-a54567fcdbc3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2fd45ad5b0146e3a38df4963fe31520'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:48.777075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1248/2531] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:48 2022] DELETE /identity/v3/users/a2fd45ad5b0146e3a38df4963fe31520 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:48.948697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702605b7-d517-4f0a-9d63-b90298e79717 service placement] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:48.959322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702605b7-d517-4f0a-9d63-b90298e79717 service placement] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:49.019234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:49.019859 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:49.020172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:49.020478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] PATH_INFO: `/v3/projects/140b03e3006e4633b718455b69e1edd6` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:49.021916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] RBAC: Authorizing `identity:delete_project(project_id=140b03e3006e4633b718455b69e1edd6)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:49.023807 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:49.072156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '140b03e3006e4633b718455b69e1edd6', 'request_id': 'req-79ac2e2a-affe-4e01-bd50-8893e37a6f29'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.077967 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '140b03e3006e4633b718455b69e1edd6', 'request_id': 'req-79ac2e2a-affe-4e01-bd50-8893e37a6f29'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.085728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '140b03e3006e4633b718455b69e1edd6', 'request_id': 'req-79ac2e2a-affe-4e01-bd50-8893e37a6f29'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.097030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] The token cache is being invalidate because project 140b03e3006e4633b718455b69e1edd6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:49.097397 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.098181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.099644 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1284/2532] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:48 2022] DELETE /identity/v3/projects/140b03e3006e4633b718455b69e1edd6 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:49.103572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363bc380-5ba2-4472-bdbe-a54567fcdbc3 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:49.104977 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-363bc380-5ba2-4472-bdbe-a54567fcdbc3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:49.118136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363bc380-5ba2-4472-bdbe-a54567fcdbc3 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:49.172967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96221d75-fb07-4e94-844a-a6f84b9e25bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:49.173595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96221d75-fb07-4e94-844a-a6f84b9e25bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:49.173882 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96221d75-fb07-4e94-844a-a6f84b9e25bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:49.174164 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96221d75-fb07-4e94-844a-a6f84b9e25bc admin admin] PATH_INFO: `/v3/users/c7b5a974044648e4ab5f7706f1622b32` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:49.175268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96221d75-fb07-4e94-844a-a6f84b9e25bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=c7b5a974044648e4ab5f7706f1622b32)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:49.177260 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96221d75-fb07-4e94-844a-a6f84b9e25bc admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:49.223228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-96221d75-fb07-4e94-844a-a6f84b9e25bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c7b5a974044648e4ab5f7706f1622b32'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.223874 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-96221d75-fb07-4e94-844a-a6f84b9e25bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c7b5a974044648e4ab5f7706f1622b32'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.224514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-96221d75-fb07-4e94-844a-a6f84b9e25bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c7b5a974044648e4ab5f7706f1622b32'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.226470 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:49.234129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-96221d75-fb07-4e94-844a-a6f84b9e25bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c7b5a974044648e4ab5f7706f1622b32'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.246976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-96221d75-fb07-4e94-844a-a6f84b9e25bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c7b5a974044648e4ab5f7706f1622b32'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.259928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1249/2533] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:49 2022] DELETE /identity/v3/users/c7b5a974044648e4ab5f7706f1622b32 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:49.431292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:49.432763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:49.443213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:49.447618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ac2e2a-affe-4e01-bd50-8893e37a6f29 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:49.507208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:49.507838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:49.508119 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:49.508428 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc admin admin] PATH_INFO: `/v3/projects/40521248b9c94502aaa5ca21d56ec6db` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:49.509577 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc admin admin] RBAC: Authorizing `identity:delete_project(project_id=40521248b9c94502aaa5ca21d56ec6db)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:49.511522 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:49.556194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40521248b9c94502aaa5ca21d56ec6db', 'request_id': 'req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.561710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40521248b9c94502aaa5ca21d56ec6db', 'request_id': 'req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.570088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40521248b9c94502aaa5ca21d56ec6db', 'request_id': 'req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.580936 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc admin admin] The token cache is being invalidate because project 40521248b9c94502aaa5ca21d56ec6db was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:49.581222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.581847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8208c988-f9fb-42a4-bdf4-bc331d4ce0bc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:49.583320 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1285/2534] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:49 2022] DELETE /identity/v3/projects/40521248b9c94502aaa5ca21d56ec6db => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:49.588201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f67ae06-dacd-4909-80c1-874100c42389 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:49.588523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f67ae06-dacd-4909-80c1-874100c42389 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:49.588801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f67ae06-dacd-4909-80c1-874100c42389 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:49.608341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-5f67ae06-dacd-4909-80c1-874100c42389 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:49.609536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5f67ae06-dacd-4909-80c1-874100c42389 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:49.667703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1250/2535] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:49.733442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c885191-c578-48ea-8bf3-86778663c220 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:49.733795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c885191-c578-48ea-8bf3-86778663c220 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:49.734154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c885191-c578-48ea-8bf3-86778663c220 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:49.760998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-0c885191-c578-48ea-8bf3-86778663c220 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:49.762614 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0c885191-c578-48ea-8bf3-86778663c220 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:49.773479 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1286/2536] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:49.777858 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f67ae06-dacd-4909-80c1-874100c42389 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:49.787856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f67ae06-dacd-4909-80c1-874100c42389 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:49.792597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52614889-5f04-4b68-b7c9-0f21b3ed880a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:49.793190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52614889-5f04-4b68-b7c9-0f21b3ed880a admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:49.793474 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52614889-5f04-4b68-b7c9-0f21b3ed880a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:49.793748 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52614889-5f04-4b68-b7c9-0f21b3ed880a admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:49.794423 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52614889-5f04-4b68-b7c9-0f21b3ed880a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:48:49.794728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52614889-5f04-4b68-b7c9-0f21b3ed880a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:49.796459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52614889-5f04-4b68-b7c9-0f21b3ed880a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:49.807784 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1251/2537] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:48:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:48:49.811860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c885191-c578-48ea-8bf3-86778663c220 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:49.821439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c885191-c578-48ea-8bf3-86778663c220 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:49.826255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde978fa-bc96-42f6-9112-e276c9e6b3ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:49.826886 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde978fa-bc96-42f6-9112-e276c9e6b3ce admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:49.827161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde978fa-bc96-42f6-9112-e276c9e6b3ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:49.827433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde978fa-bc96-42f6-9112-e276c9e6b3ce admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:49.828071 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde978fa-bc96-42f6-9112-e276c9e6b3ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:49.830007 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde978fa-bc96-42f6-9112-e276c9e6b3ce admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:49.857040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1287/2538] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:49 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:49.861365 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52614889-5f04-4b68-b7c9-0f21b3ed880a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:49.871191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52614889-5f04-4b68-b7c9-0f21b3ed880a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:49.927064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f5c5e9-8afd-45d8-902e-dc8bf559ad69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:49.927662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f5c5e9-8afd-45d8-902e-dc8bf559ad69 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:49.927949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f5c5e9-8afd-45d8-902e-dc8bf559ad69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:49.928228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f5c5e9-8afd-45d8-902e-dc8bf559ad69 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:49.928895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f5c5e9-8afd-45d8-902e-dc8bf559ad69 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:49.930841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f5c5e9-8afd-45d8-902e-dc8bf559ad69 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:49.951397 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1252/2539] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:49 2022] POST /identity/v3/users => generated 287 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:49.955510 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde978fa-bc96-42f6-9112-e276c9e6b3ce admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:49.966270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde978fa-bc96-42f6-9112-e276c9e6b3ce admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:49.970886 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80598ad-f28a-4fa2-8a67-8956201078a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:49.971475 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80598ad-f28a-4fa2-8a67-8956201078a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:49.971762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80598ad-f28a-4fa2-8a67-8956201078a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:49.972038 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80598ad-f28a-4fa2-8a67-8956201078a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:49.972747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80598ad-f28a-4fa2-8a67-8956201078a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:49.974913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80598ad-f28a-4fa2-8a67-8956201078a0 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:49.984703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1288/2540] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:49.988756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f5c5e9-8afd-45d8-902e-dc8bf559ad69 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:49.998689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f5c5e9-8afd-45d8-902e-dc8bf559ad69 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:50.003259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca56d7c-551c-478c-9fd8-1816e83afa1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:50.003845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca56d7c-551c-478c-9fd8-1816e83afa1e admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:50.004127 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca56d7c-551c-478c-9fd8-1816e83afa1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:50.004459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca56d7c-551c-478c-9fd8-1816e83afa1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:50.005105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca56d7c-551c-478c-9fd8-1816e83afa1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:50.006702 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca56d7c-551c-478c-9fd8-1816e83afa1e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:50.017254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1253/2541] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:49 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:50.021957 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80598ad-f28a-4fa2-8a67-8956201078a0 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:50.036945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80598ad-f28a-4fa2-8a67-8956201078a0 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:50.042532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6b8d95-21d7-45b6-949c-3b37c8af50b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:50.043186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6b8d95-21d7-45b6-949c-3b37c8af50b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:50.043468 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6b8d95-21d7-45b6-949c-3b37c8af50b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:50.043742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6b8d95-21d7-45b6-949c-3b37c8af50b6 admin admin] PATH_INFO: `/v3/projects/20cabe0d6ea8489ea2ee86865cc99af7/users/0ef5e246ad1c4abf8e3b0a6684ff5b5e/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:50.061788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6b8d95-21d7-45b6-949c-3b37c8af50b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=20cabe0d6ea8489ea2ee86865cc99af7, user_id=0ef5e246ad1c4abf8e3b0a6684ff5b5e, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:50.064467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6b8d95-21d7-45b6-949c-3b37c8af50b6 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:50.078399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1289/2542] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:50 2022] PUT /identity/v3/projects/20cabe0d6ea8489ea2ee86865cc99af7/users/0ef5e246ad1c4abf8e3b0a6684ff5b5e/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:50.088700 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c13857-8abe-4dd1-910a-a311bdf4c029 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:50.088986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c13857-8abe-4dd1-910a-a311bdf4c029 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:50.089283 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c13857-8abe-4dd1-910a-a311bdf4c029 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:50.117756 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-07c13857-8abe-4dd1-910a-a311bdf4c029 None None] MFA Rules not processed for user `0ef5e246ad1c4abf8e3b0a6684ff5b5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:50.118964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-07c13857-8abe-4dd1-910a-a311bdf4c029 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:50.138727 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:50.184902 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1254/2543] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:50 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:48:50.189353 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6b8d95-21d7-45b6-949c-3b37c8af50b6 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:50.224025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:50.255181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6b8d95-21d7-45b6-949c-3b37c8af50b6 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:50.259823 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcab6a27-14fd-408a-b43a-883adc44dccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:50.260448 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcab6a27-14fd-408a-b43a-883adc44dccc admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:50.260730 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcab6a27-14fd-408a-b43a-883adc44dccc admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:50.261002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcab6a27-14fd-408a-b43a-883adc44dccc admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:50.261641 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcab6a27-14fd-408a-b43a-883adc44dccc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:50.263525 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcab6a27-14fd-408a-b43a-883adc44dccc admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:50.288894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1290/2544] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:50 2022] POST /identity/v3/projects => generated 371 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:48:50.293063 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c13857-8abe-4dd1-910a-a311bdf4c029 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:50.304639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c13857-8abe-4dd1-910a-a311bdf4c029 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:50.364245 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e2b0d1-965a-4a11-a3e4-22e2a6d6903c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:50.364899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e2b0d1-965a-4a11-a3e4-22e2a6d6903c admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:50.365179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e2b0d1-965a-4a11-a3e4-22e2a6d6903c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:50.365452 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e2b0d1-965a-4a11-a3e4-22e2a6d6903c admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:50.366089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e2b0d1-965a-4a11-a3e4-22e2a6d6903c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:50.368041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e2b0d1-965a-4a11-a3e4-22e2a6d6903c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:50.390309 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1255/2545] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:50 2022] POST /identity/v3/users => generated 292 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:50.394876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcab6a27-14fd-408a-b43a-883adc44dccc admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:50.407445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcab6a27-14fd-408a-b43a-883adc44dccc admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:50.415222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4879555f-4453-40c9-9da9-4d7732467099 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:50.415992 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4879555f-4453-40c9-9da9-4d7732467099 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:50.416283 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4879555f-4453-40c9-9da9-4d7732467099 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:50.416839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4879555f-4453-40c9-9da9-4d7732467099 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:50.417418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4879555f-4453-40c9-9da9-4d7732467099 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:50.419050 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4879555f-4453-40c9-9da9-4d7732467099 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:50.439696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1291/2546] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:50 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:50.442227 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e2b0d1-965a-4a11-a3e4-22e2a6d6903c admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:50.452896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e2b0d1-965a-4a11-a3e4-22e2a6d6903c admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:50.457728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5658ad7f-0be0-4836-9dc2-daa39c6d02fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:50.458379 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5658ad7f-0be0-4836-9dc2-daa39c6d02fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:50.458662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5658ad7f-0be0-4836-9dc2-daa39c6d02fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:50.458937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5658ad7f-0be0-4836-9dc2-daa39c6d02fa admin admin] PATH_INFO: `/v3/projects/15c29683f080481f99037c2ce6d1d6cd/users/0af2ce3930b84e0aabc40e57a6d7f0da/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:50.474437 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5658ad7f-0be0-4836-9dc2-daa39c6d02fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=15c29683f080481f99037c2ce6d1d6cd, user_id=0af2ce3930b84e0aabc40e57a6d7f0da, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:50.477190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5658ad7f-0be0-4836-9dc2-daa39c6d02fa admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:50.496879 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1256/2547] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:50 2022] PUT /identity/v3/projects/15c29683f080481f99037c2ce6d1d6cd/users/0af2ce3930b84e0aabc40e57a6d7f0da/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:50.501040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4879555f-4453-40c9-9da9-4d7732467099 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:50.513346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4879555f-4453-40c9-9da9-4d7732467099 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:50.569901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb63d1f-a950-4a1a-b35d-1e066d142c1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:50.570432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb63d1f-a950-4a1a-b35d-1e066d142c1a admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:50.570714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb63d1f-a950-4a1a-b35d-1e066d142c1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:50.570989 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb63d1f-a950-4a1a-b35d-1e066d142c1a admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:50.571632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb63d1f-a950-4a1a-b35d-1e066d142c1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:50.573265 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb63d1f-a950-4a1a-b35d-1e066d142c1a admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:50.583688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1292/2548] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:50 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:50.587822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5658ad7f-0be0-4836-9dc2-daa39c6d02fa admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:50.598131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5658ad7f-0be0-4836-9dc2-daa39c6d02fa admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:50.603034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4160d652-2484-4c51-bb5d-6d78867b102a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:50.603676 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4160d652-2484-4c51-bb5d-6d78867b102a admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:50.603952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4160d652-2484-4c51-bb5d-6d78867b102a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:50.604226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4160d652-2484-4c51-bb5d-6d78867b102a admin admin] PATH_INFO: `/v3/projects/15c29683f080481f99037c2ce6d1d6cd/users/0af2ce3930b84e0aabc40e57a6d7f0da/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:50.606200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4160d652-2484-4c51-bb5d-6d78867b102a admin admin] RBAC: Authorizing `identity:create_grant(project_id=15c29683f080481f99037c2ce6d1d6cd, user_id=0af2ce3930b84e0aabc40e57a6d7f0da, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:50.608877 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4160d652-2484-4c51-bb5d-6d78867b102a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:50.619883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1257/2549] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:50 2022] PUT /identity/v3/projects/15c29683f080481f99037c2ce6d1d6cd/users/0af2ce3930b84e0aabc40e57a6d7f0da/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:50.630868 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f830c7-abed-4390-82a0-a7a91588926f None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:50.631188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f830c7-abed-4390-82a0-a7a91588926f None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:50.631470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f830c7-abed-4390-82a0-a7a91588926f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:50.660029 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-a6f830c7-abed-4390-82a0-a7a91588926f None None] MFA Rules not processed for user `0af2ce3930b84e0aabc40e57a6d7f0da`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:50.661977 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a6f830c7-abed-4390-82a0-a7a91588926f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:50.742411 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1293/2550] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:50 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:50.756842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4160d652-2484-4c51-bb5d-6d78867b102a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:50.758903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-4160d652-2484-4c51-bb5d-6d78867b102a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:50.832903 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4160d652-2484-4c51-bb5d-6d78867b102a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:50.837083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31d5509-543f-48c4-9cb8-7dcbe2c04ac1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:50.837754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31d5509-543f-48c4-9cb8-7dcbe2c04ac1 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:50.838030 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31d5509-543f-48c4-9cb8-7dcbe2c04ac1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:50.838294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31d5509-543f-48c4-9cb8-7dcbe2c04ac1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:50.847164 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31d5509-543f-48c4-9cb8-7dcbe2c04ac1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:50.848623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31d5509-543f-48c4-9cb8-7dcbe2c04ac1 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:50.858332 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1258/2551] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:50 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:51.262529 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f830c7-abed-4390-82a0-a7a91588926f None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:51.293826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:51.326769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f830c7-abed-4390-82a0-a7a91588926f None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:51.331400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a228d2-5a5d-4aef-b317-6c9217b85bc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:51.332023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a228d2-5a5d-4aef-b317-6c9217b85bc4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:51.332297 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a228d2-5a5d-4aef-b317-6c9217b85bc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:51.332624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a228d2-5a5d-4aef-b317-6c9217b85bc4 admin admin] PATH_INFO: `/v3/users/0ef5e246ad1c4abf8e3b0a6684ff5b5e` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:51.333729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a228d2-5a5d-4aef-b317-6c9217b85bc4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0ef5e246ad1c4abf8e3b0a6684ff5b5e)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:51.335658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a228d2-5a5d-4aef-b317-6c9217b85bc4 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:51.383523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-83a228d2-5a5d-4aef-b317-6c9217b85bc4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0ef5e246ad1c4abf8e3b0a6684ff5b5e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:51.384177 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-83a228d2-5a5d-4aef-b317-6c9217b85bc4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0ef5e246ad1c4abf8e3b0a6684ff5b5e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:51.391439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-83a228d2-5a5d-4aef-b317-6c9217b85bc4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0ef5e246ad1c4abf8e3b0a6684ff5b5e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:51.392164 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-83a228d2-5a5d-4aef-b317-6c9217b85bc4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0ef5e246ad1c4abf8e3b0a6684ff5b5e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:51.404903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-83a228d2-5a5d-4aef-b317-6c9217b85bc4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0ef5e246ad1c4abf8e3b0a6684ff5b5e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:51.420332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1294/2552] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:51 2022] DELETE /identity/v3/users/0ef5e246ad1c4abf8e3b0a6684ff5b5e => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:51.424450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde29834-5b10-4e3c-b909-6ebe105fb0bc None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:51.424749 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde29834-5b10-4e3c-b909-6ebe105fb0bc None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:51.425095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde29834-5b10-4e3c-b909-6ebe105fb0bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:51.444133 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-dde29834-5b10-4e3c-b909-6ebe105fb0bc None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:51.445350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-dde29834-5b10-4e3c-b909-6ebe105fb0bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:51.456618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:51.507919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1259/2553] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:51.517707 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a228d2-5a5d-4aef-b317-6c9217b85bc4 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:51.519106 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-83a228d2-5a5d-4aef-b317-6c9217b85bc4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:51.575990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a228d2-5a5d-4aef-b317-6c9217b85bc4 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:51.579660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75189cdf-cb86-4921-81e3-d642d86f81d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:51.580194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75189cdf-cb86-4921-81e3-d642d86f81d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:51.580500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75189cdf-cb86-4921-81e3-d642d86f81d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:51.580773 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75189cdf-cb86-4921-81e3-d642d86f81d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:51.590022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75189cdf-cb86-4921-81e3-d642d86f81d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:51.591431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75189cdf-cb86-4921-81e3-d642d86f81d6 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:51.601238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1295/2554] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:51.862700 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde29834-5b10-4e3c-b909-6ebe105fb0bc None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:51.865226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-dde29834-5b10-4e3c-b909-6ebe105fb0bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:51.886914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde29834-5b10-4e3c-b909-6ebe105fb0bc None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:51.894207 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:51.894780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:51.895615 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:51.895875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] PATH_INFO: `/v3/projects/20cabe0d6ea8489ea2ee86865cc99af7` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:51.897132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] RBAC: Authorizing `identity:delete_project(project_id=20cabe0d6ea8489ea2ee86865cc99af7)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:51.899047 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:51.949095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '20cabe0d6ea8489ea2ee86865cc99af7', 'request_id': 'req-60c33722-3d67-4dd6-aff5-b0e6bca28a40'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:51.955056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '20cabe0d6ea8489ea2ee86865cc99af7', 'request_id': 'req-60c33722-3d67-4dd6-aff5-b0e6bca28a40'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:51.963864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '20cabe0d6ea8489ea2ee86865cc99af7', 'request_id': 'req-60c33722-3d67-4dd6-aff5-b0e6bca28a40'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:51.976857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] The token cache is being invalidate because project 20cabe0d6ea8489ea2ee86865cc99af7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:51.977219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:51.977875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:51.979412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1260/2555] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:51 2022] DELETE /identity/v3/projects/20cabe0d6ea8489ea2ee86865cc99af7 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:51.985769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75189cdf-cb86-4921-81e3-d642d86f81d6 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:51.996092 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75189cdf-cb86-4921-81e3-d642d86f81d6 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:52.040265 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:52.053217 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6341f15c-5751-464c-ace9-aa4a8534b684 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:52.053834 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6341f15c-5751-464c-ace9-aa4a8534b684 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:52.054111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6341f15c-5751-464c-ace9-aa4a8534b684 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:52.054382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6341f15c-5751-464c-ace9-aa4a8534b684 admin admin] PATH_INFO: `/v3/users/0af2ce3930b84e0aabc40e57a6d7f0da` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:52.055492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6341f15c-5751-464c-ace9-aa4a8534b684 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0af2ce3930b84e0aabc40e57a6d7f0da)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:52.057435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6341f15c-5751-464c-ace9-aa4a8534b684 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:52.105929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6341f15c-5751-464c-ace9-aa4a8534b684 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0af2ce3930b84e0aabc40e57a6d7f0da'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:52.106887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6341f15c-5751-464c-ace9-aa4a8534b684 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0af2ce3930b84e0aabc40e57a6d7f0da'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:52.115844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6341f15c-5751-464c-ace9-aa4a8534b684 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0af2ce3930b84e0aabc40e57a6d7f0da'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:52.116550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6341f15c-5751-464c-ace9-aa4a8534b684 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0af2ce3930b84e0aabc40e57a6d7f0da'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:52.130383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6341f15c-5751-464c-ace9-aa4a8534b684 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0af2ce3930b84e0aabc40e57a6d7f0da'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:52.150439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1296/2556] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:51 2022] DELETE /identity/v3/users/0af2ce3930b84e0aabc40e57a6d7f0da => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:52.262220 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:52.263639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:52.346319 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c33722-3d67-4dd6-aff5-b0e6bca28a40 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:52.397812 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7913e0-0ac5-48b5-9fe5-28a42a2ff4b6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:52.398905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7913e0-0ac5-48b5-9fe5-28a42a2ff4b6 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:52.398905 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7913e0-0ac5-48b5-9fe5-28a42a2ff4b6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:52.399083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7913e0-0ac5-48b5-9fe5-28a42a2ff4b6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:52.401597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0a7913e0-0ac5-48b5-9fe5-28a42a2ff4b6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:52.413389 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:52.500776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7913e0-0ac5-48b5-9fe5-28a42a2ff4b6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:52.502766 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7913e0-0ac5-48b5-9fe5-28a42a2ff4b6 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:52.514338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6341f15c-5751-464c-ace9-aa4a8534b684 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:52.516181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6341f15c-5751-464c-ace9-aa4a8534b684 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:52.527938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1261/2557] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:52 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:52.606553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6341f15c-5751-464c-ace9-aa4a8534b684 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:52.615067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b8ba27-ee09-4e9f-8227-c0e7396347e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:52.615713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b8ba27-ee09-4e9f-8227-c0e7396347e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:52.615996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b8ba27-ee09-4e9f-8227-c0e7396347e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:52.616261 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b8ba27-ee09-4e9f-8227-c0e7396347e7 admin admin] PATH_INFO: `/v3/projects/15c29683f080481f99037c2ce6d1d6cd` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:52.617502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b8ba27-ee09-4e9f-8227-c0e7396347e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=15c29683f080481f99037c2ce6d1d6cd)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:52.619400 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b8ba27-ee09-4e9f-8227-c0e7396347e7 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:52.700134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-38b8ba27-ee09-4e9f-8227-c0e7396347e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15c29683f080481f99037c2ce6d1d6cd', 'request_id': 'req-38b8ba27-ee09-4e9f-8227-c0e7396347e7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:52.706277 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-38b8ba27-ee09-4e9f-8227-c0e7396347e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15c29683f080481f99037c2ce6d1d6cd', 'request_id': 'req-38b8ba27-ee09-4e9f-8227-c0e7396347e7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:52.714658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-38b8ba27-ee09-4e9f-8227-c0e7396347e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15c29683f080481f99037c2ce6d1d6cd', 'request_id': 'req-38b8ba27-ee09-4e9f-8227-c0e7396347e7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:52.725141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:52.730474 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-38b8ba27-ee09-4e9f-8227-c0e7396347e7 admin admin] The token cache is being invalidate because project 15c29683f080481f99037c2ce6d1d6cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:52.730761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-38b8ba27-ee09-4e9f-8227-c0e7396347e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:52.731655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-38b8ba27-ee09-4e9f-8227-c0e7396347e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:52.736493 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1297/2558] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:52 2022] DELETE /identity/v3/projects/15c29683f080481f99037c2ce6d1d6cd => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:52.739434 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b277fc55-68a7-4dce-a8d0-ee70f0c1d3af None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:52.739757 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b277fc55-68a7-4dce-a8d0-ee70f0c1d3af None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:52.740129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b277fc55-68a7-4dce-a8d0-ee70f0c1d3af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:52.773274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-b277fc55-68a7-4dce-a8d0-ee70f0c1d3af None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:52.776462 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b277fc55-68a7-4dce-a8d0-ee70f0c1d3af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:52.851570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1262/2559] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:48:52.859069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cf122d-47f1-4dc7-9d5a-714e1cded978 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:52.859360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cf122d-47f1-4dc7-9d5a-714e1cded978 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:52.859642 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cf122d-47f1-4dc7-9d5a-714e1cded978 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:52.885116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-91cf122d-47f1-4dc7-9d5a-714e1cded978 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:52.886404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-91cf122d-47f1-4dc7-9d5a-714e1cded978 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:52.894719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1298/2560] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:52.899774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b277fc55-68a7-4dce-a8d0-ee70f0c1d3af None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:52.916115 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b277fc55-68a7-4dce-a8d0-ee70f0c1d3af None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:52.921193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a8d32d-b16f-4901-9d8e-e1c532540e62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:52.921804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a8d32d-b16f-4901-9d8e-e1c532540e62 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:52.922084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a8d32d-b16f-4901-9d8e-e1c532540e62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:52.922573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a8d32d-b16f-4901-9d8e-e1c532540e62 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:52.923287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a8d32d-b16f-4901-9d8e-e1c532540e62 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:48:52.923544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a8d32d-b16f-4901-9d8e-e1c532540e62 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:52.925218 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a8d32d-b16f-4901-9d8e-e1c532540e62 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:52.937952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1263/2561] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:48:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:48:52.943130 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cf122d-47f1-4dc7-9d5a-714e1cded978 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:52.952510 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cf122d-47f1-4dc7-9d5a-714e1cded978 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:52.964657 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68cda08-0415-4464-a7f9-a6472c7cec42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:52.965284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68cda08-0415-4464-a7f9-a6472c7cec42 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:52.965565 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68cda08-0415-4464-a7f9-a6472c7cec42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:52.965838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68cda08-0415-4464-a7f9-a6472c7cec42 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:52.966489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68cda08-0415-4464-a7f9-a6472c7cec42 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:52.968625 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68cda08-0415-4464-a7f9-a6472c7cec42 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:53.001766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1299/2562] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:52 2022] POST /identity/v3/projects => generated 379 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:53.005896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a8d32d-b16f-4901-9d8e-e1c532540e62 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:53.016993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a8d32d-b16f-4901-9d8e-e1c532540e62 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:53.090060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9017610c-f327-4ea6-aa5c-63f4d3158c95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:53.090670 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9017610c-f327-4ea6-aa5c-63f4d3158c95 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:53.091039 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9017610c-f327-4ea6-aa5c-63f4d3158c95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:53.091319 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9017610c-f327-4ea6-aa5c-63f4d3158c95 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:53.091952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9017610c-f327-4ea6-aa5c-63f4d3158c95 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:53.094123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9017610c-f327-4ea6-aa5c-63f4d3158c95 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:53.118842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1264/2563] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:53 2022] POST /identity/v3/users => generated 290 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:53.123309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68cda08-0415-4464-a7f9-a6472c7cec42 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:53.135509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68cda08-0415-4464-a7f9-a6472c7cec42 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:53.141970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044a637d-111d-41ad-8db5-66e7efc213a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:53.142894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044a637d-111d-41ad-8db5-66e7efc213a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:53.143246 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044a637d-111d-41ad-8db5-66e7efc213a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:53.143590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044a637d-111d-41ad-8db5-66e7efc213a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:53.144405 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044a637d-111d-41ad-8db5-66e7efc213a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:53.146579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044a637d-111d-41ad-8db5-66e7efc213a9 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:53.160492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1300/2564] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:53 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:48:53.162505 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9017610c-f327-4ea6-aa5c-63f4d3158c95 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:53.172827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9017610c-f327-4ea6-aa5c-63f4d3158c95 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:53.178500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6169cd-86ef-4135-8b48-efee511ad7e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:53.179096 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6169cd-86ef-4135-8b48-efee511ad7e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:53.179370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6169cd-86ef-4135-8b48-efee511ad7e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:53.179638 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6169cd-86ef-4135-8b48-efee511ad7e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:53.180277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6169cd-86ef-4135-8b48-efee511ad7e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:53.181952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6169cd-86ef-4135-8b48-efee511ad7e2 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:53.196474 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1265/2565] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:53 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:53.200701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044a637d-111d-41ad-8db5-66e7efc213a9 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:53.212205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044a637d-111d-41ad-8db5-66e7efc213a9 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:53.217009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bea356e-29b7-477c-bdba-92cdac14baac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:53.217651 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bea356e-29b7-477c-bdba-92cdac14baac admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:53.217927 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bea356e-29b7-477c-bdba-92cdac14baac admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:53.218197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bea356e-29b7-477c-bdba-92cdac14baac admin admin] PATH_INFO: `/v3/projects/90b49460fbe64ef493bd9a3205c21024/users/c002f77a6d3845aeb7e34ab86baf76c7/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:53.235373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bea356e-29b7-477c-bdba-92cdac14baac admin admin] RBAC: Authorizing `identity:create_grant(project_id=90b49460fbe64ef493bd9a3205c21024, user_id=c002f77a6d3845aeb7e34ab86baf76c7, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:53.238610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bea356e-29b7-477c-bdba-92cdac14baac admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:53.253262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1301/2566] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:53 2022] PUT /identity/v3/projects/90b49460fbe64ef493bd9a3205c21024/users/c002f77a6d3845aeb7e34ab86baf76c7/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:53.263928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a4befe-1a01-4c2e-97f0-3811752c1177 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:53.264209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a4befe-1a01-4c2e-97f0-3811752c1177 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:53.264511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a4befe-1a01-4c2e-97f0-3811752c1177 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:53.289318 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:53.293390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-41a4befe-1a01-4c2e-97f0-3811752c1177 None None] MFA Rules not processed for user `c002f77a6d3845aeb7e34ab86baf76c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:53.295258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-41a4befe-1a01-4c2e-97f0-3811752c1177 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:53.361566 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1266/2567] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:53 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:53.366214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bea356e-29b7-477c-bdba-92cdac14baac admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:53.430552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bea356e-29b7-477c-bdba-92cdac14baac admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:53.435626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb645e1-f0af-4b44-8847-8d29ed828615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:53.436509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb645e1-f0af-4b44-8847-8d29ed828615 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:53.436805 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb645e1-f0af-4b44-8847-8d29ed828615 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:53.437080 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb645e1-f0af-4b44-8847-8d29ed828615 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:53.437933 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb645e1-f0af-4b44-8847-8d29ed828615 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:53.440066 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb645e1-f0af-4b44-8847-8d29ed828615 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:53.468365 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1302/2568] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:53 2022] POST /identity/v3/projects => generated 379 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:53.472504 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a4befe-1a01-4c2e-97f0-3811752c1177 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:53.484013 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a4befe-1a01-4c2e-97f0-3811752c1177 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:53.544992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e729f84b-4b56-4217-ad03-ed1e3f734755 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:53.545618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e729f84b-4b56-4217-ad03-ed1e3f734755 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:53.545888 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e729f84b-4b56-4217-ad03-ed1e3f734755 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:53.546163 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e729f84b-4b56-4217-ad03-ed1e3f734755 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:53.546804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e729f84b-4b56-4217-ad03-ed1e3f734755 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:53.548794 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e729f84b-4b56-4217-ad03-ed1e3f734755 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:53.570483 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1267/2569] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:53 2022] POST /identity/v3/users => generated 296 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:53.574446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb645e1-f0af-4b44-8847-8d29ed828615 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:53.585255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb645e1-f0af-4b44-8847-8d29ed828615 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:53.589813 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea24d88-2eeb-4470-8401-92dbcb875dfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:53.590406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea24d88-2eeb-4470-8401-92dbcb875dfa admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:53.590686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea24d88-2eeb-4470-8401-92dbcb875dfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:53.591003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea24d88-2eeb-4470-8401-92dbcb875dfa admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:53.591677 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea24d88-2eeb-4470-8401-92dbcb875dfa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:53.593288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea24d88-2eeb-4470-8401-92dbcb875dfa admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:53.603648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1303/2570] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:53.607710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e729f84b-4b56-4217-ad03-ed1e3f734755 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:53.618178 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e729f84b-4b56-4217-ad03-ed1e3f734755 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:53.622764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335f9dd0-770a-4d8f-a4f5-129cfa710390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:53.623406 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335f9dd0-770a-4d8f-a4f5-129cfa710390 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:53.623688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335f9dd0-770a-4d8f-a4f5-129cfa710390 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:53.623965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335f9dd0-770a-4d8f-a4f5-129cfa710390 admin admin] PATH_INFO: `/v3/projects/37207ef6a8734a7c93be89a753249bf6/users/ba13837adc0c4f9b811aede1ce6c64b8/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:53.638818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335f9dd0-770a-4d8f-a4f5-129cfa710390 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37207ef6a8734a7c93be89a753249bf6, user_id=ba13837adc0c4f9b811aede1ce6c64b8, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:53.641519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335f9dd0-770a-4d8f-a4f5-129cfa710390 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:53.654691 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1268/2571] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:53 2022] PUT /identity/v3/projects/37207ef6a8734a7c93be89a753249bf6/users/ba13837adc0c4f9b811aede1ce6c64b8/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:53.658497 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea24d88-2eeb-4470-8401-92dbcb875dfa admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:53.668513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea24d88-2eeb-4470-8401-92dbcb875dfa admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:53.724989 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab9f8bd-9daf-4583-8332-e3c84215b1af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:53.725590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab9f8bd-9daf-4583-8332-e3c84215b1af admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:53.726152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab9f8bd-9daf-4583-8332-e3c84215b1af admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:53.726502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab9f8bd-9daf-4583-8332-e3c84215b1af admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:53.727138 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab9f8bd-9daf-4583-8332-e3c84215b1af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:53.728744 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab9f8bd-9daf-4583-8332-e3c84215b1af admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:53.739021 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1304/2572] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:53 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:53.743086 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335f9dd0-770a-4d8f-a4f5-129cfa710390 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:53.753131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335f9dd0-770a-4d8f-a4f5-129cfa710390 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:53.758255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbfc8b4-7c25-40c3-9a57-a52a532ad187 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:53.758882 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbfc8b4-7c25-40c3-9a57-a52a532ad187 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:53.759158 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbfc8b4-7c25-40c3-9a57-a52a532ad187 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:53.759427 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbfc8b4-7c25-40c3-9a57-a52a532ad187 admin admin] PATH_INFO: `/v3/projects/37207ef6a8734a7c93be89a753249bf6/users/ba13837adc0c4f9b811aede1ce6c64b8/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:53.761354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbfc8b4-7c25-40c3-9a57-a52a532ad187 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37207ef6a8734a7c93be89a753249bf6, user_id=ba13837adc0c4f9b811aede1ce6c64b8, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:53.763956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbfc8b4-7c25-40c3-9a57-a52a532ad187 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:53.773840 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1269/2573] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:53 2022] PUT /identity/v3/projects/37207ef6a8734a7c93be89a753249bf6/users/ba13837adc0c4f9b811aede1ce6c64b8/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:53.784591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3a1cf2-40d8-48a6-a585-82a290971843 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:53.784879 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3a1cf2-40d8-48a6-a585-82a290971843 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:53.785153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3a1cf2-40d8-48a6-a585-82a290971843 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:53.793316 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:53.815391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-9f3a1cf2-40d8-48a6-a585-82a290971843 None None] MFA Rules not processed for user `ba13837adc0c4f9b811aede1ce6c64b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:53.816659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-9f3a1cf2-40d8-48a6-a585-82a290971843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:53.886758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1305/2574] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:53 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:53.899696 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbfc8b4-7c25-40c3-9a57-a52a532ad187 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:53.901450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-4cbfc8b4-7c25-40c3-9a57-a52a532ad187 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:53.960025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbfc8b4-7c25-40c3-9a57-a52a532ad187 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:53.963956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff0f65d-8594-44f9-9ebf-cd4f0005db06 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:53.964639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff0f65d-8594-44f9-9ebf-cd4f0005db06 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:53.964936 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff0f65d-8594-44f9-9ebf-cd4f0005db06 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:53.965218 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff0f65d-8594-44f9-9ebf-cd4f0005db06 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:53.974950 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff0f65d-8594-44f9-9ebf-cd4f0005db06 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:53.976599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff0f65d-8594-44f9-9ebf-cd4f0005db06 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:53.980624 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:53.989425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1270/2575] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:53 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:54.353182 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3a1cf2-40d8-48a6-a585-82a290971843 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:54.362620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3a1cf2-40d8-48a6-a585-82a290971843 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:54.367251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488fd060-d878-4a37-b56b-7dc7323dc3d6 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba13837adc0c4f9b811aede1ce6c64b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '37207ef6a8734a7c93be89a753249bf6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:54.367852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488fd060-d878-4a37-b56b-7dc7323dc3d6 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:54.368144 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488fd060-d878-4a37-b56b-7dc7323dc3d6 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:54.368461 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488fd060-d878-4a37-b56b-7dc7323dc3d6 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:54.369117 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488fd060-d878-4a37-b56b-7dc7323dc3d6 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:48:54.369427 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488fd060-d878-4a37-b56b-7dc7323dc3d6 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:54.371014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488fd060-d878-4a37-b56b-7dc7323dc3d6 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:54.383040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1306/2576] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:48:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:48:54.387123 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff0f65d-8594-44f9-9ebf-cd4f0005db06 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:54.396649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff0f65d-8594-44f9-9ebf-cd4f0005db06 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:54.401329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921fc988-0fac-49a3-a84b-21c14f112564 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba13837adc0c4f9b811aede1ce6c64b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '37207ef6a8734a7c93be89a753249bf6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:54.401969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921fc988-0fac-49a3-a84b-21c14f112564 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:54.402261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921fc988-0fac-49a3-a84b-21c14f112564 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:54.402529 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921fc988-0fac-49a3-a84b-21c14f112564 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:54.403172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921fc988-0fac-49a3-a84b-21c14f112564 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:54.405145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921fc988-0fac-49a3-a84b-21c14f112564 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:54.437736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1271/2577] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:54 2022] POST /identity/v3/projects => generated 354 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:54.438147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488fd060-d878-4a37-b56b-7dc7323dc3d6 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:54.438705 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921fc988-0fac-49a3-a84b-21c14f112564 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:54.439932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-488fd060-d878-4a37-b56b-7dc7323dc3d6 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:54.504499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921fc988-0fac-49a3-a84b-21c14f112564 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:54.509361 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df2a30e-9d8d-4efd-9882-603ccde1ba5b tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba13837adc0c4f9b811aede1ce6c64b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '37207ef6a8734a7c93be89a753249bf6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:54.509963 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2a30e-9d8d-4efd-9882-603ccde1ba5b tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:54.510244 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2a30e-9d8d-4efd-9882-603ccde1ba5b tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:54.510521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2a30e-9d8d-4efd-9882-603ccde1ba5b tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:54.511165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df2a30e-9d8d-4efd-9882-603ccde1ba5b tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:48:54.511736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df2a30e-9d8d-4efd-9882-603ccde1ba5b tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:54.513387 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df2a30e-9d8d-4efd-9882-603ccde1ba5b tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:54.516677 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488fd060-d878-4a37-b56b-7dc7323dc3d6 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:54.525506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1272/2578] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:48:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:48:54.568590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df2a30e-9d8d-4efd-9882-603ccde1ba5b tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:54.579930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df2a30e-9d8d-4efd-9882-603ccde1ba5b tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:54.580141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbb367b-d301-4b3a-afe7-1d9dc9b14536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:54.580694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbb367b-d301-4b3a-afe7-1d9dc9b14536 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:54.580975 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbb367b-d301-4b3a-afe7-1d9dc9b14536 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:54.581243 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbb367b-d301-4b3a-afe7-1d9dc9b14536 admin admin] PATH_INFO: `/v3/users/0430169e743e4d00a49c5d24b0d48df7` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:54.582459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbb367b-d301-4b3a-afe7-1d9dc9b14536 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0430169e743e4d00a49c5d24b0d48df7)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:54.584433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbb367b-d301-4b3a-afe7-1d9dc9b14536 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:54.585906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ba13837adc0c4f9b811aede1ce6c64b8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '37207ef6a8734a7c93be89a753249bf6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:54.588185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:54.588718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:54.589194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] PATH_INFO: `/v3/projects/542ebf9b7fa348348837b50f183428a3` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:54.590602 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] RBAC: Authorizing `identity:delete_project(project_id=542ebf9b7fa348348837b50f183428a3)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:54.592521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:54.621900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:54.646696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ecbb367b-d301-4b3a-afe7-1d9dc9b14536 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0430169e743e4d00a49c5d24b0d48df7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:54.647633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ecbb367b-d301-4b3a-afe7-1d9dc9b14536 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0430169e743e4d00a49c5d24b0d48df7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:54.655723 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ecbb367b-d301-4b3a-afe7-1d9dc9b14536 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0430169e743e4d00a49c5d24b0d48df7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:54.656409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ecbb367b-d301-4b3a-afe7-1d9dc9b14536 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0430169e743e4d00a49c5d24b0d48df7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:54.658845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '542ebf9b7fa348348837b50f183428a3', 'request_id': 'req-cf497b22-1784-4598-b0ab-dd4f04c749a8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:54.666581 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '542ebf9b7fa348348837b50f183428a3', 'request_id': 'req-cf497b22-1784-4598-b0ab-dd4f04c749a8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:54.669734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-ecbb367b-d301-4b3a-afe7-1d9dc9b14536 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0430169e743e4d00a49c5d24b0d48df7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:54.679231 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '542ebf9b7fa348348837b50f183428a3', 'request_id': 'req-cf497b22-1784-4598-b0ab-dd4f04c749a8'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:54.688250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1307/2579] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:54 2022] DELETE /identity/v3/users/0430169e743e4d00a49c5d24b0d48df7 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:54.694149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75294f63-9327-4b64-8a7f-08dba5c4f6cd None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:54.694149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75294f63-9327-4b64-8a7f-08dba5c4f6cd None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:54.694149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75294f63-9327-4b64-8a7f-08dba5c4f6cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:54.696764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] The token cache is being invalidate because project 542ebf9b7fa348348837b50f183428a3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:54.697179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:54.698232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:54.700281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1273/2580] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:54 2022] DELETE /identity/v3/projects/542ebf9b7fa348348837b50f183428a3 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:54.718111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-75294f63-9327-4b64-8a7f-08dba5c4f6cd None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:54.719377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-75294f63-9327-4b64-8a7f-08dba5c4f6cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:54.794630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1308/2581] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:48:54.805536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:54.807170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:54.874770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf497b22-1784-4598-b0ab-dd4f04c749a8 tempest-VolumeQuotaClassesTest-648717622 tempest-VolumeQuotaClassesTest-648717622-project-admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:54.880982 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:54.914491 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75294f63-9327-4b64-8a7f-08dba5c4f6cd None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:54.915900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-75294f63-9327-4b64-8a7f-08dba5c4f6cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:54.928539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75294f63-9327-4b64-8a7f-08dba5c4f6cd None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:54.933215 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a9b722-da29-40e3-8c38-eebaffcc7124 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:54.933719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddacc0e-06d7-4605-8b1c-f4fbaaba1329 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:54.933828 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a9b722-da29-40e3-8c38-eebaffcc7124 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:54.934062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a9b722-da29-40e3-8c38-eebaffcc7124 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:54.934343 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a9b722-da29-40e3-8c38-eebaffcc7124 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:54.934487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddacc0e-06d7-4605-8b1c-f4fbaaba1329 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:54.934759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddacc0e-06d7-4605-8b1c-f4fbaaba1329 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:54.935107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddacc0e-06d7-4605-8b1c-f4fbaaba1329 admin admin] PATH_INFO: `/v3/users/c002f77a6d3845aeb7e34ab86baf76c7` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:54.936662 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eddacc0e-06d7-4605-8b1c-f4fbaaba1329 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c002f77a6d3845aeb7e34ab86baf76c7)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:54.938870 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eddacc0e-06d7-4605-8b1c-f4fbaaba1329 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:54.944170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a9b722-da29-40e3-8c38-eebaffcc7124 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:54.945989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a9b722-da29-40e3-8c38-eebaffcc7124 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:54.955695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1274/2582] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:54.989621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-eddacc0e-06d7-4605-8b1c-f4fbaaba1329 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c002f77a6d3845aeb7e34ab86baf76c7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:54.990357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-eddacc0e-06d7-4605-8b1c-f4fbaaba1329 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c002f77a6d3845aeb7e34ab86baf76c7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:54.998091 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-eddacc0e-06d7-4605-8b1c-f4fbaaba1329 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c002f77a6d3845aeb7e34ab86baf76c7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:54.998762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-eddacc0e-06d7-4605-8b1c-f4fbaaba1329 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c002f77a6d3845aeb7e34ab86baf76c7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.012435 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-eddacc0e-06d7-4605-8b1c-f4fbaaba1329 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c002f77a6d3845aeb7e34ab86baf76c7'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.026546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1309/2583] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:54 2022] DELETE /identity/v3/users/c002f77a6d3845aeb7e34ab86baf76c7 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:55.031199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00445d13-4004-42f8-8d7f-49c4d9684d48 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:55.031506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00445d13-4004-42f8-8d7f-49c4d9684d48 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:55.031852 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00445d13-4004-42f8-8d7f-49c4d9684d48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:55.052015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-00445d13-4004-42f8-8d7f-49c4d9684d48 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:55.053244 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-00445d13-4004-42f8-8d7f-49c4d9684d48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:55.116697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1275/2584] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:55.127231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddacc0e-06d7-4605-8b1c-f4fbaaba1329 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:55.128955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-eddacc0e-06d7-4605-8b1c-f4fbaaba1329 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:55.174893 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:55.225089 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddacc0e-06d7-4605-8b1c-f4fbaaba1329 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:55.229555 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1387cb8-1349-4bef-8c0f-62d5e4edb76f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:55.230355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1387cb8-1349-4bef-8c0f-62d5e4edb76f service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:55.230734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1387cb8-1349-4bef-8c0f-62d5e4edb76f service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:55.231107 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1387cb8-1349-4bef-8c0f-62d5e4edb76f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:55.242672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00445d13-4004-42f8-8d7f-49c4d9684d48 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:55.244144 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-00445d13-4004-42f8-8d7f-49c4d9684d48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:55.258479 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00445d13-4004-42f8-8d7f-49c4d9684d48 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:55.259290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1387cb8-1349-4bef-8c0f-62d5e4edb76f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:55.260806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1387cb8-1349-4bef-8c0f-62d5e4edb76f service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:55.263843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:55.264743 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:55.264995 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:55.265378 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] PATH_INFO: `/v3/projects/56ed67dbbfdd4159925f9a25b37d52f4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:55.266826 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=56ed67dbbfdd4159925f9a25b37d52f4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:55.268842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:55.275976 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1310/2585] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:55.339931 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56ed67dbbfdd4159925f9a25b37d52f4', 'request_id': 'req-48fde95e-e02a-498d-9a3c-d97690698bc9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.346266 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56ed67dbbfdd4159925f9a25b37d52f4', 'request_id': 'req-48fde95e-e02a-498d-9a3c-d97690698bc9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.355926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56ed67dbbfdd4159925f9a25b37d52f4', 'request_id': 'req-48fde95e-e02a-498d-9a3c-d97690698bc9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.373036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] The token cache is being invalidate because project 56ed67dbbfdd4159925f9a25b37d52f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:55.373036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.373324 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.375050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1276/2586] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:55 2022] DELETE /identity/v3/projects/56ed67dbbfdd4159925f9a25b37d52f4 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:55.379356 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1387cb8-1349-4bef-8c0f-62d5e4edb76f service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:55.392551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1387cb8-1349-4bef-8c0f-62d5e4edb76f service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:55.468526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2ab9f1-10e4-47df-8830-ab9fb50485b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:55.468526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2ab9f1-10e4-47df-8830-ab9fb50485b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:55.468526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2ab9f1-10e4-47df-8830-ab9fb50485b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:55.468526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2ab9f1-10e4-47df-8830-ab9fb50485b3 admin admin] PATH_INFO: `/v3/users/880fc2d17e5c405daf719c589234413a` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:55.469731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e2ab9f1-10e4-47df-8830-ab9fb50485b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=880fc2d17e5c405daf719c589234413a)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:55.471625 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e2ab9f1-10e4-47df-8830-ab9fb50485b3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:55.529622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7e2ab9f1-10e4-47df-8830-ab9fb50485b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '880fc2d17e5c405daf719c589234413a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.530302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7e2ab9f1-10e4-47df-8830-ab9fb50485b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '880fc2d17e5c405daf719c589234413a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.538367 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7e2ab9f1-10e4-47df-8830-ab9fb50485b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '880fc2d17e5c405daf719c589234413a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.539044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7e2ab9f1-10e4-47df-8830-ab9fb50485b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '880fc2d17e5c405daf719c589234413a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.551478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7e2ab9f1-10e4-47df-8830-ab9fb50485b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '880fc2d17e5c405daf719c589234413a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.567609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1311/2587] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:55 2022] DELETE /identity/v3/users/880fc2d17e5c405daf719c589234413a => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:55.638793 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:55.640725 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:55.657652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fde95e-e02a-498d-9a3c-d97690698bc9 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:55.696060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:55.727489 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:55.728153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:55.728514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:55.728790 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] PATH_INFO: `/v3/projects/90b49460fbe64ef493bd9a3205c21024` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:55.730007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=90b49460fbe64ef493bd9a3205c21024)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:55.732098 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:55.787822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '90b49460fbe64ef493bd9a3205c21024', 'request_id': 'req-aab52d43-4faa-4757-a3d7-de63b55363cd'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.796372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '90b49460fbe64ef493bd9a3205c21024', 'request_id': 'req-aab52d43-4faa-4757-a3d7-de63b55363cd'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.806896 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '90b49460fbe64ef493bd9a3205c21024', 'request_id': 'req-aab52d43-4faa-4757-a3d7-de63b55363cd'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.819706 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] The token cache is being invalidate because project 90b49460fbe64ef493bd9a3205c21024 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:55.820010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.820656 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.822446 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1277/2588] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:55 2022] DELETE /identity/v3/projects/90b49460fbe64ef493bd9a3205c21024 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:55.826521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2ab9f1-10e4-47df-8830-ab9fb50485b3 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:55.827979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7e2ab9f1-10e4-47df-8830-ab9fb50485b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:55.840975 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2ab9f1-10e4-47df-8830-ab9fb50485b3 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:55.874574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:55.876062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:55.877741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:55.920541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac9b86c-f246-4308-aea1-71b120d2280e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:55.921134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac9b86c-f246-4308-aea1-71b120d2280e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:55.921832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac9b86c-f246-4308-aea1-71b120d2280e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:55.922131 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac9b86c-f246-4308-aea1-71b120d2280e admin admin] PATH_INFO: `/v3/users/ba13837adc0c4f9b811aede1ce6c64b8` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:55.923619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac9b86c-f246-4308-aea1-71b120d2280e admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba13837adc0c4f9b811aede1ce6c64b8)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:55.925576 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac9b86c-f246-4308-aea1-71b120d2280e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:55.946171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab52d43-4faa-4757-a3d7-de63b55363cd admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:55.952081 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b5984f-c65b-4d75-8522-57b55f095b2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:55.952860 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b5984f-c65b-4d75-8522-57b55f095b2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:55.953141 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b5984f-c65b-4d75-8522-57b55f095b2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:55.953438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b5984f-c65b-4d75-8522-57b55f095b2a admin admin] PATH_INFO: `/v3/projects/7aa98378563f40fda40166fd622af528` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:55.954853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b5984f-c65b-4d75-8522-57b55f095b2a admin admin] RBAC: Authorizing `identity:delete_project(project_id=7aa98378563f40fda40166fd622af528)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:55.957940 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b5984f-c65b-4d75-8522-57b55f095b2a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:55.972695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-cac9b86c-f246-4308-aea1-71b120d2280e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba13837adc0c4f9b811aede1ce6c64b8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.973403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-cac9b86c-f246-4308-aea1-71b120d2280e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba13837adc0c4f9b811aede1ce6c64b8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.980939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-cac9b86c-f246-4308-aea1-71b120d2280e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba13837adc0c4f9b811aede1ce6c64b8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.981599 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-cac9b86c-f246-4308-aea1-71b120d2280e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba13837adc0c4f9b811aede1ce6c64b8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:55.994864 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-cac9b86c-f246-4308-aea1-71b120d2280e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba13837adc0c4f9b811aede1ce6c64b8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:56.007980 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1312/2589] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:55 2022] DELETE /identity/v3/users/ba13837adc0c4f9b811aede1ce6c64b8 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:56.010593 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-08b5984f-c65b-4d75-8522-57b55f095b2a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7aa98378563f40fda40166fd622af528', 'request_id': 'req-08b5984f-c65b-4d75-8522-57b55f095b2a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:56.016516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-08b5984f-c65b-4d75-8522-57b55f095b2a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7aa98378563f40fda40166fd622af528', 'request_id': 'req-08b5984f-c65b-4d75-8522-57b55f095b2a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:56.024568 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-08b5984f-c65b-4d75-8522-57b55f095b2a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7aa98378563f40fda40166fd622af528', 'request_id': 'req-08b5984f-c65b-4d75-8522-57b55f095b2a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:56.026774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:56.038935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-08b5984f-c65b-4d75-8522-57b55f095b2a admin admin] The token cache is being invalidate because project 7aa98378563f40fda40166fd622af528 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:56.039248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-08b5984f-c65b-4d75-8522-57b55f095b2a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:56.040496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-08b5984f-c65b-4d75-8522-57b55f095b2a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:56.042267 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1278/2590] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:55 2022] DELETE /identity/v3/projects/7aa98378563f40fda40166fd622af528 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:56.047095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1c9a7c-8be4-4318-8f41-152bd24b05c9 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.047383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1c9a7c-8be4-4318-8f41-152bd24b05c9 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.047668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1c9a7c-8be4-4318-8f41-152bd24b05c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.068923 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4b1c9a7c-8be4-4318-8f41-152bd24b05c9 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:56.070191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4b1c9a7c-8be4-4318-8f41-152bd24b05c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:56.134584 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1313/2591] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:48:56.142053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7456bf2c-04c2-4ee8-b642-fa661d7b8370 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.142340 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7456bf2c-04c2-4ee8-b642-fa661d7b8370 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.142612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7456bf2c-04c2-4ee8-b642-fa661d7b8370 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.169000 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-7456bf2c-04c2-4ee8-b642-fa661d7b8370 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:56.170403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-7456bf2c-04c2-4ee8-b642-fa661d7b8370 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:56.179027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1279/2592] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:56.183180 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1c9a7c-8be4-4318-8f41-152bd24b05c9 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:56.193292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1c9a7c-8be4-4318-8f41-152bd24b05c9 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:56.197983 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b15e906-8a6e-48e0-8356-d28b24293846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:56.198576 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b15e906-8a6e-48e0-8356-d28b24293846 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.198854 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b15e906-8a6e-48e0-8356-d28b24293846 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.199123 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b15e906-8a6e-48e0-8356-d28b24293846 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.199763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b15e906-8a6e-48e0-8356-d28b24293846 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:48:56.200064 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b15e906-8a6e-48e0-8356-d28b24293846 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:56.201663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b15e906-8a6e-48e0-8356-d28b24293846 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:56.213520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1314/2593] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:48:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:48:56.217711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7456bf2c-04c2-4ee8-b642-fa661d7b8370 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:56.228031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7456bf2c-04c2-4ee8-b642-fa661d7b8370 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:56.232995 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cf7a4f-b9f9-4f99-af8b-be1406102095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:56.233590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cf7a4f-b9f9-4f99-af8b-be1406102095 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.233869 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cf7a4f-b9f9-4f99-af8b-be1406102095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.234141 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cf7a4f-b9f9-4f99-af8b-be1406102095 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.234999 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cf7a4f-b9f9-4f99-af8b-be1406102095 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:56.236983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cf7a4f-b9f9-4f99-af8b-be1406102095 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:56.266862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1280/2594] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:56 2022] POST /identity/v3/projects => generated 385 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:56.272050 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b15e906-8a6e-48e0-8356-d28b24293846 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:56.283716 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b15e906-8a6e-48e0-8356-d28b24293846 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:56.288714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cf7a4f-b9f9-4f99-af8b-be1406102095 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:56.290378 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-26cf7a4f-b9f9-4f99-af8b-be1406102095 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:56.336239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:56.354336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4376cf-c692-41e4-b16c-46a9e29ce801 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:56.355642 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4376cf-c692-41e4-b16c-46a9e29ce801 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.355927 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4376cf-c692-41e4-b16c-46a9e29ce801 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.356402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4376cf-c692-41e4-b16c-46a9e29ce801 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.357066 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4376cf-c692-41e4-b16c-46a9e29ce801 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:56.358396 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cf7a4f-b9f9-4f99-af8b-be1406102095 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:56.359046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4376cf-c692-41e4-b16c-46a9e29ce801 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:56.363293 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:56.364504 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.364784 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.365061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba admin admin] PATH_INFO: `/v3/projects/37207ef6a8734a7c93be89a753249bf6` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.366221 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=37207ef6a8734a7c93be89a753249bf6)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:56.368133 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:56.380722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1315/2595] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:56 2022] POST /identity/v3/users => generated 293 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:56.384657 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4376cf-c692-41e4-b16c-46a9e29ce801 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:56.394001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4376cf-c692-41e4-b16c-46a9e29ce801 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:56.398681 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eaf62f-6e91-44e2-a165-67e99861467d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:56.399293 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eaf62f-6e91-44e2-a165-67e99861467d admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.399649 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eaf62f-6e91-44e2-a165-67e99861467d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.399937 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eaf62f-6e91-44e2-a165-67e99861467d admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.400777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9eaf62f-6e91-44e2-a165-67e99861467d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:56.402540 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9eaf62f-6e91-44e2-a165-67e99861467d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:56.417340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1316/2596] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:48:56.418666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eaf62f-6e91-44e2-a165-67e99861467d admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:56.423300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37207ef6a8734a7c93be89a753249bf6', 'request_id': 'req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:56.429252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eaf62f-6e91-44e2-a165-67e99861467d admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:56.429493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37207ef6a8734a7c93be89a753249bf6', 'request_id': 'req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:56.438726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37207ef6a8734a7c93be89a753249bf6', 'request_id': 'req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:56.455048 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba admin admin] The token cache is being invalidate because project 37207ef6a8734a7c93be89a753249bf6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:56.455342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:56.456022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-3fd6e74f-d1c0-4f26-ab40-bc2a96f338ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:56.457575 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1281/2597] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:56 2022] DELETE /identity/v3/projects/37207ef6a8734a7c93be89a753249bf6 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:56.462145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc23e988-83e6-4180-a07c-6014167f39c9 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.462381 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc23e988-83e6-4180-a07c-6014167f39c9 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.462655 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc23e988-83e6-4180-a07c-6014167f39c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.489948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-dc23e988-83e6-4180-a07c-6014167f39c9 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:56.491239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-dc23e988-83e6-4180-a07c-6014167f39c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:56.514140 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34264ee-bafc-4d69-8f44-98c89324bf9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:56.514924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34264ee-bafc-4d69-8f44-98c89324bf9b admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.515452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34264ee-bafc-4d69-8f44-98c89324bf9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.515739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34264ee-bafc-4d69-8f44-98c89324bf9b admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.516514 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34264ee-bafc-4d69-8f44-98c89324bf9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:56.518220 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34264ee-bafc-4d69-8f44-98c89324bf9b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:56.530155 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1317/2598] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:56 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:48:56.534388 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34264ee-bafc-4d69-8f44-98c89324bf9b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:56.537444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c34264ee-bafc-4d69-8f44-98c89324bf9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:56.552578 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34264ee-bafc-4d69-8f44-98c89324bf9b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:56.557778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792f7cb0-1c31-4f14-b642-427d86b0171b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:56.558669 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:56.559062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792f7cb0-1c31-4f14-b642-427d86b0171b admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.559548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792f7cb0-1c31-4f14-b642-427d86b0171b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.559926 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792f7cb0-1c31-4f14-b642-427d86b0171b admin admin] PATH_INFO: `/v3/projects/bc1ed9ade340401b89103e12238dd8fa/users/0f1d2780dee3431fa59fe505f64c9ba2/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.576981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1282/2599] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:56.582884 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66270f79-3bae-4c88-a102-0bb8a4ce2d50 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.583245 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66270f79-3bae-4c88-a102-0bb8a4ce2d50 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.583666 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66270f79-3bae-4c88-a102-0bb8a4ce2d50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.584502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792f7cb0-1c31-4f14-b642-427d86b0171b admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc1ed9ade340401b89103e12238dd8fa, user_id=0f1d2780dee3431fa59fe505f64c9ba2, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:56.587647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792f7cb0-1c31-4f14-b642-427d86b0171b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:56.607584 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-66270f79-3bae-4c88-a102-0bb8a4ce2d50 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:56.608777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1318/2600] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:56 2022] PUT /identity/v3/projects/bc1ed9ade340401b89103e12238dd8fa/users/0f1d2780dee3431fa59fe505f64c9ba2/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:56.619862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-66270f79-3bae-4c88-a102-0bb8a4ce2d50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:56.621421 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a8f994-323f-4eda-91e0-e773e2e4bd96 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.621776 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a8f994-323f-4eda-91e0-e773e2e4bd96 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.622043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a8f994-323f-4eda-91e0-e773e2e4bd96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.650179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-40a8f994-323f-4eda-91e0-e773e2e4bd96 None None] MFA Rules not processed for user `0f1d2780dee3431fa59fe505f64c9ba2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:56.651498 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-40a8f994-323f-4eda-91e0-e773e2e4bd96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:56.688050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1283/2601] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:56.698044 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66270f79-3bae-4c88-a102-0bb8a4ce2d50 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:56.709356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66270f79-3bae-4c88-a102-0bb8a4ce2d50 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:56.717187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0460f4ab-c480-4cc3-bd3b-a0395110e6a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:56.717911 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0460f4ab-c480-4cc3-bd3b-a0395110e6a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.718302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0460f4ab-c480-4cc3-bd3b-a0395110e6a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.718697 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0460f4ab-c480-4cc3-bd3b-a0395110e6a2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.720019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0460f4ab-c480-4cc3-bd3b-a0395110e6a2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:48:56.720503 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0460f4ab-c480-4cc3-bd3b-a0395110e6a2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:56.723767 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0460f4ab-c480-4cc3-bd3b-a0395110e6a2 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:56.725132 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1319/2602] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:56 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:48:56.729590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a8f994-323f-4eda-91e0-e773e2e4bd96 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:56.743002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1284/2603] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:48:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:48:56.746955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0460f4ab-c480-4cc3-bd3b-a0395110e6a2 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:56.747911 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a8f994-323f-4eda-91e0-e773e2e4bd96 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:56.753075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab98eee-33ea-43bd-811f-9d5039b2e17c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:56.753719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab98eee-33ea-43bd-811f-9d5039b2e17c admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.754118 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab98eee-33ea-43bd-811f-9d5039b2e17c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.754366 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab98eee-33ea-43bd-811f-9d5039b2e17c admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.755361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab98eee-33ea-43bd-811f-9d5039b2e17c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:56.758148 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab98eee-33ea-43bd-811f-9d5039b2e17c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:56.758275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0460f4ab-c480-4cc3-bd3b-a0395110e6a2 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:56.787548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903a4649-0e44-40fd-8859-439ce6d4e698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:56.787548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903a4649-0e44-40fd-8859-439ce6d4e698 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.787548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903a4649-0e44-40fd-8859-439ce6d4e698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.787548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903a4649-0e44-40fd-8859-439ce6d4e698 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.788007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903a4649-0e44-40fd-8859-439ce6d4e698 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:56.788007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903a4649-0e44-40fd-8859-439ce6d4e698 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:56.804851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1320/2604] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:56 2022] POST /identity/v3/projects => generated 385 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:48:56.808541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab98eee-33ea-43bd-811f-9d5039b2e17c admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:56.812699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1285/2605] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:56 2022] POST /identity/v3/projects => generated 381 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:48:56.814177 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:56.816625 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903a4649-0e44-40fd-8859-439ce6d4e698 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:56.907011 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab98eee-33ea-43bd-811f-9d5039b2e17c admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:56.923731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903a4649-0e44-40fd-8859-439ce6d4e698 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:56.929026 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ef2278-3f97-4309-b167-5fca9aaf151c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:56.929622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ef2278-3f97-4309-b167-5fca9aaf151c admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.929951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ef2278-3f97-4309-b167-5fca9aaf151c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.930167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ef2278-3f97-4309-b167-5fca9aaf151c admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.930796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ef2278-3f97-4309-b167-5fca9aaf151c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:56.932762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ef2278-3f97-4309-b167-5fca9aaf151c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:56.954076 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1286/2606] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:56 2022] POST /identity/v3/users => generated 291 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:48:56.959354 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ef2278-3f97-4309-b167-5fca9aaf151c admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:56.968605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d428f11-bfd1-40fa-beea-80cc620ea79e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:56.969094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ef2278-3f97-4309-b167-5fca9aaf151c admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:56.969211 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d428f11-bfd1-40fa-beea-80cc620ea79e admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.969472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d428f11-bfd1-40fa-beea-80cc620ea79e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.969743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d428f11-bfd1-40fa-beea-80cc620ea79e admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.970381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d428f11-bfd1-40fa-beea-80cc620ea79e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:56.972263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d428f11-bfd1-40fa-beea-80cc620ea79e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:56.974468 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae661b-1349-4540-b51d-7ff29c7c7825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:56.975157 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae661b-1349-4540-b51d-7ff29c7c7825 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:56.975442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae661b-1349-4540-b51d-7ff29c7c7825 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:56.975717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae661b-1349-4540-b51d-7ff29c7c7825 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:56.976380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae661b-1349-4540-b51d-7ff29c7c7825 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:56.977983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae661b-1349-4540-b51d-7ff29c7c7825 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:56.992489 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1287/2607] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:48:56.993414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae661b-1349-4540-b51d-7ff29c7c7825 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.000906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1321/2608] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:56 2022] POST /identity/v3/users => generated 299 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:48:57.001906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d428f11-bfd1-40fa-beea-80cc620ea79e admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.003125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae661b-1349-4540-b51d-7ff29c7c7825 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.008464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f892f085-b6f1-477a-8a68-231cffe5d703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.009075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f892f085-b6f1-477a-8a68-231cffe5d703 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.009424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f892f085-b6f1-477a-8a68-231cffe5d703 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.009639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f892f085-b6f1-477a-8a68-231cffe5d703 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.010266 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f892f085-b6f1-477a-8a68-231cffe5d703 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.011698 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d428f11-bfd1-40fa-beea-80cc620ea79e admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.011864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f892f085-b6f1-477a-8a68-231cffe5d703 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.016766 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c4397c-6fba-4827-8450-9b8f4307633e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.017814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c4397c-6fba-4827-8450-9b8f4307633e admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.018079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c4397c-6fba-4827-8450-9b8f4307633e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.018347 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c4397c-6fba-4827-8450-9b8f4307633e admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.018979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c4397c-6fba-4827-8450-9b8f4307633e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.021120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c4397c-6fba-4827-8450-9b8f4307633e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.023487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1288/2609] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:48:57.027471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f892f085-b6f1-477a-8a68-231cffe5d703 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.032276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1322/2610] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:57.036973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c4397c-6fba-4827-8450-9b8f4307633e admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.037785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f892f085-b6f1-477a-8a68-231cffe5d703 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.042751 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29344106-36fe-487e-811e-1176cbce5d00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.043382 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29344106-36fe-487e-811e-1176cbce5d00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.043662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29344106-36fe-487e-811e-1176cbce5d00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.043926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29344106-36fe-487e-811e-1176cbce5d00 admin admin] PATH_INFO: `/v3/projects/61dccc193bc443c5afe1ae3c2dcab74a/users/dc4fe8bad42047e2827e37e8cac76a75/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.046702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c4397c-6fba-4827-8450-9b8f4307633e admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.051271 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116235d6-e7c0-4b8c-ae47-d5aa35a62ced admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.051891 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116235d6-e7c0-4b8c-ae47-d5aa35a62ced admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.052167 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116235d6-e7c0-4b8c-ae47-d5aa35a62ced admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.052473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116235d6-e7c0-4b8c-ae47-d5aa35a62ced admin admin] PATH_INFO: `/v3/projects/26150a2d9bc1446390e11492aaf2eff5/users/a1cb51d8f5c945d2aaa12ec585374210/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.058148 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29344106-36fe-487e-811e-1176cbce5d00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=61dccc193bc443c5afe1ae3c2dcab74a, user_id=dc4fe8bad42047e2827e37e8cac76a75, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.060827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29344106-36fe-487e-811e-1176cbce5d00 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.068046 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116235d6-e7c0-4b8c-ae47-d5aa35a62ced admin admin] RBAC: Authorizing `identity:create_grant(project_id=26150a2d9bc1446390e11492aaf2eff5, user_id=a1cb51d8f5c945d2aaa12ec585374210, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.070777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116235d6-e7c0-4b8c-ae47-d5aa35a62ced admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.074388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1289/2611] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:57 2022] PUT /identity/v3/projects/61dccc193bc443c5afe1ae3c2dcab74a/users/dc4fe8bad42047e2827e37e8cac76a75/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:57.084425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1323/2612] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:57 2022] PUT /identity/v3/projects/26150a2d9bc1446390e11492aaf2eff5/users/a1cb51d8f5c945d2aaa12ec585374210/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:57.084886 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c389cb1-0eda-4a20-821b-f28d4e8dfdb5 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.085173 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c389cb1-0eda-4a20-821b-f28d4e8dfdb5 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.085449 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c389cb1-0eda-4a20-821b-f28d4e8dfdb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.088197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116235d6-e7c0-4b8c-ae47-d5aa35a62ced admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.098547 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:57.114177 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-5c389cb1-0eda-4a20-821b-f28d4e8dfdb5 None None] MFA Rules not processed for user `dc4fe8bad42047e2827e37e8cac76a75`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:57.115515 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5c389cb1-0eda-4a20-821b-f28d4e8dfdb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:57.155313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116235d6-e7c0-4b8c-ae47-d5aa35a62ced admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.160130 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab94ff0-6e6c-4339-b7a7-f384094b3120 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.160789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab94ff0-6e6c-4339-b7a7-f384094b3120 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.161069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab94ff0-6e6c-4339-b7a7-f384094b3120 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.161372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab94ff0-6e6c-4339-b7a7-f384094b3120 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.162017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab94ff0-6e6c-4339-b7a7-f384094b3120 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.163603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab94ff0-6e6c-4339-b7a7-f384094b3120 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.168913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:57.174374 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1324/2613] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:57 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:57.178507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab94ff0-6e6c-4339-b7a7-f384094b3120 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.185914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1290/2614] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:57 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:48:57.190150 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c389cb1-0eda-4a20-821b-f28d4e8dfdb5 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.190887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab94ff0-6e6c-4339-b7a7-f384094b3120 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.195951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68e7077-cc62-4c6c-ad15-ccb69f52150f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.196758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68e7077-cc62-4c6c-ad15-ccb69f52150f admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.197147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68e7077-cc62-4c6c-ad15-ccb69f52150f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.197427 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68e7077-cc62-4c6c-ad15-ccb69f52150f admin admin] PATH_INFO: `/v3/projects/26150a2d9bc1446390e11492aaf2eff5/users/a1cb51d8f5c945d2aaa12ec585374210/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.199351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68e7077-cc62-4c6c-ad15-ccb69f52150f admin admin] RBAC: Authorizing `identity:create_grant(project_id=26150a2d9bc1446390e11492aaf2eff5, user_id=a1cb51d8f5c945d2aaa12ec585374210, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.200736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c389cb1-0eda-4a20-821b-f28d4e8dfdb5 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.202001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68e7077-cc62-4c6c-ad15-ccb69f52150f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.205392 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83601a9-dba7-43b6-bf36-7575156555f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.205969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83601a9-dba7-43b6-bf36-7575156555f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.206249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83601a9-dba7-43b6-bf36-7575156555f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.206516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83601a9-dba7-43b6-bf36-7575156555f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.207131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83601a9-dba7-43b6-bf36-7575156555f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.209113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83601a9-dba7-43b6-bf36-7575156555f7 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.212554 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1325/2615] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:57 2022] PUT /identity/v3/projects/26150a2d9bc1446390e11492aaf2eff5/users/a1cb51d8f5c945d2aaa12ec585374210/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:57.222717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70adf971-5326-4e6d-9bb4-a0452aa41a9b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.223547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70adf971-5326-4e6d-9bb4-a0452aa41a9b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.223843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70adf971-5326-4e6d-9bb4-a0452aa41a9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.236855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1291/2616] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:57 2022] POST /identity/v3/projects => generated 383 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:48:57.241236 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83601a9-dba7-43b6-bf36-7575156555f7 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.253461 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-70adf971-5326-4e6d-9bb4-a0452aa41a9b None None] MFA Rules not processed for user `a1cb51d8f5c945d2aaa12ec585374210`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:57.254733 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-70adf971-5326-4e6d-9bb4-a0452aa41a9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:57.302242 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83601a9-dba7-43b6-bf36-7575156555f7 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.307269 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf429f4-509c-4069-ab03-3190b5c9327c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.307887 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf429f4-509c-4069-ab03-3190b5c9327c admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.308158 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf429f4-509c-4069-ab03-3190b5c9327c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.308458 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf429f4-509c-4069-ab03-3190b5c9327c admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.309074 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf429f4-509c-4069-ab03-3190b5c9327c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.310985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf429f4-509c-4069-ab03-3190b5c9327c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.325189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1326/2617] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:57 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:48:57.335535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1292/2618] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:57 2022] POST /identity/v3/users => generated 298 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:48:57.336362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70adf971-5326-4e6d-9bb4-a0452aa41a9b None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.336631 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf429f4-509c-4069-ab03-3190b5c9327c admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.337716 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-70adf971-5326-4e6d-9bb4-a0452aa41a9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:57.347364 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf429f4-509c-4069-ab03-3190b5c9327c admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.352934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c6b162-a91f-43a2-8e7f-e4daab25e23e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.353521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c6b162-a91f-43a2-8e7f-e4daab25e23e admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.353800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c6b162-a91f-43a2-8e7f-e4daab25e23e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.354075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c6b162-a91f-43a2-8e7f-e4daab25e23e admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.354713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c6b162-a91f-43a2-8e7f-e4daab25e23e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.356352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c6b162-a91f-43a2-8e7f-e4daab25e23e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.368011 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1293/2619] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:48:57.372229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c6b162-a91f-43a2-8e7f-e4daab25e23e admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.382745 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c6b162-a91f-43a2-8e7f-e4daab25e23e admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.387342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36af9d5-18af-45ae-9ac2-cb91d2d12ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.387969 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36af9d5-18af-45ae-9ac2-cb91d2d12ee7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.388246 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36af9d5-18af-45ae-9ac2-cb91d2d12ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.388562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36af9d5-18af-45ae-9ac2-cb91d2d12ee7 admin admin] PATH_INFO: `/v3/projects/c71e1834bba241f487ddc6832200e0ef/users/aa41c9d3e7c44642a058aa17ad51fe33/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.394490 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70adf971-5326-4e6d-9bb4-a0452aa41a9b None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.398103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03afd4a5-b65f-4319-ac99-5eefdd890e01 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.398695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03afd4a5-b65f-4319-ac99-5eefdd890e01 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.398978 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03afd4a5-b65f-4319-ac99-5eefdd890e01 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.399239 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03afd4a5-b65f-4319-ac99-5eefdd890e01 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.404248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36af9d5-18af-45ae-9ac2-cb91d2d12ee7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c71e1834bba241f487ddc6832200e0ef, user_id=aa41c9d3e7c44642a058aa17ad51fe33, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.406928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36af9d5-18af-45ae-9ac2-cb91d2d12ee7 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.408281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03afd4a5-b65f-4319-ac99-5eefdd890e01 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.409890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03afd4a5-b65f-4319-ac99-5eefdd890e01 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.420882 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1294/2620] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:57 2022] PUT /identity/v3/projects/c71e1834bba241f487ddc6832200e0ef/users/aa41c9d3e7c44642a058aa17ad51fe33/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:57.421019 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1327/2621] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:57 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:57.424732 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36af9d5-18af-45ae-9ac2-cb91d2d12ee7 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.456719 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:57.497618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36af9d5-18af-45ae-9ac2-cb91d2d12ee7 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.503152 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f25244e-d84c-42c5-b25e-cf7338ce723b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.503963 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f25244e-d84c-42c5-b25e-cf7338ce723b admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.504274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f25244e-d84c-42c5-b25e-cf7338ce723b admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.504583 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f25244e-d84c-42c5-b25e-cf7338ce723b admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.505284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f25244e-d84c-42c5-b25e-cf7338ce723b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.506949 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f25244e-d84c-42c5-b25e-cf7338ce723b admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.517845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1295/2622] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:48:57 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:48:57.522183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03afd4a5-b65f-4319-ac99-5eefdd890e01 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.532649 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03afd4a5-b65f-4319-ac99-5eefdd890e01 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.538077 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c229e4-357d-4674-95c6-88ea62533c1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.538724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c229e4-357d-4674-95c6-88ea62533c1f admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.539003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c229e4-357d-4674-95c6-88ea62533c1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.539273 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c229e4-357d-4674-95c6-88ea62533c1f admin admin] PATH_INFO: `/v3/projects/c71e1834bba241f487ddc6832200e0ef/users/aa41c9d3e7c44642a058aa17ad51fe33/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.541316 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c229e4-357d-4674-95c6-88ea62533c1f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c71e1834bba241f487ddc6832200e0ef, user_id=aa41c9d3e7c44642a058aa17ad51fe33, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.543931 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c229e4-357d-4674-95c6-88ea62533c1f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.554630 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1328/2623] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:48:57 2022] PUT /identity/v3/projects/c71e1834bba241f487ddc6832200e0ef/users/aa41c9d3e7c44642a058aa17ad51fe33/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:57.566001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31494275-45bd-41e1-9e77-9acc0887c2f0 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.566268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31494275-45bd-41e1-9e77-9acc0887c2f0 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.566571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31494275-45bd-41e1-9e77-9acc0887c2f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.595345 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-31494275-45bd-41e1-9e77-9acc0887c2f0 None None] MFA Rules not processed for user `aa41c9d3e7c44642a058aa17ad51fe33`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:57.596593 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-31494275-45bd-41e1-9e77-9acc0887c2f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:57.681954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1296/2624] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:57 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:57.698864 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c229e4-357d-4674-95c6-88ea62533c1f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.700885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-40c229e4-357d-4674-95c6-88ea62533c1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:57.761782 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c229e4-357d-4674-95c6-88ea62533c1f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.770994 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdde9586-8230-41fd-bb59-6ef035709848 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.771608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdde9586-8230-41fd-bb59-6ef035709848 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.771884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdde9586-8230-41fd-bb59-6ef035709848 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.772153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdde9586-8230-41fd-bb59-6ef035709848 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.781728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdde9586-8230-41fd-bb59-6ef035709848 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.783158 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdde9586-8230-41fd-bb59-6ef035709848 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.793495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1329/2625] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:57 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:57.871102 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31494275-45bd-41e1-9e77-9acc0887c2f0 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:57.881879 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31494275-45bd-41e1-9e77-9acc0887c2f0 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:57.885871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e400ffb-91f7-4f89-a7f9-c4e4a0734b57 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:57.886472 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e400ffb-91f7-4f89-a7f9-c4e4a0734b57 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:57.886861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e400ffb-91f7-4f89-a7f9-c4e4a0734b57 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:57.887094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e400ffb-91f7-4f89-a7f9-c4e4a0734b57 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:57.963458 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e400ffb-91f7-4f89-a7f9-c4e4a0734b57 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:57.963458 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e400ffb-91f7-4f89-a7f9-c4e4a0734b57 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:57.971223 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1297/2626] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:57 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:58.075395 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdde9586-8230-41fd-bb59-6ef035709848 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:58.081494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:58.158814 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdde9586-8230-41fd-bb59-6ef035709848 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:58.164653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8276d9a5-5ed3-4c32-bcc2-529349a4cfb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:58.165310 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8276d9a5-5ed3-4c32-bcc2-529349a4cfb0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:58.165542 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8276d9a5-5ed3-4c32-bcc2-529349a4cfb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:58.165809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8276d9a5-5ed3-4c32-bcc2-529349a4cfb0 admin admin] PATH_INFO: `/v3/users/0f1d2780dee3431fa59fe505f64c9ba2` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:58.172606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8276d9a5-5ed3-4c32-bcc2-529349a4cfb0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0f1d2780dee3431fa59fe505f64c9ba2)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:58.174551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8276d9a5-5ed3-4c32-bcc2-529349a4cfb0 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:58.232018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8276d9a5-5ed3-4c32-bcc2-529349a4cfb0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0f1d2780dee3431fa59fe505f64c9ba2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.236750 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8276d9a5-5ed3-4c32-bcc2-529349a4cfb0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0f1d2780dee3431fa59fe505f64c9ba2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.241808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8276d9a5-5ed3-4c32-bcc2-529349a4cfb0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0f1d2780dee3431fa59fe505f64c9ba2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.242648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8276d9a5-5ed3-4c32-bcc2-529349a4cfb0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0f1d2780dee3431fa59fe505f64c9ba2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.262744 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8276d9a5-5ed3-4c32-bcc2-529349a4cfb0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0f1d2780dee3431fa59fe505f64c9ba2'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.277997 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1330/2627] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:58 2022] DELETE /identity/v3/users/0f1d2780dee3431fa59fe505f64c9ba2 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:58.282556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f0b8c8-a43a-427d-a6bc-581753ad359c None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:58.282843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f0b8c8-a43a-427d-a6bc-581753ad359c None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:58.283114 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f0b8c8-a43a-427d-a6bc-581753ad359c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:58.304126 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-a6f0b8c8-a43a-427d-a6bc-581753ad359c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:58.305928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a6f0b8c8-a43a-427d-a6bc-581753ad359c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:58.329004 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8276d9a5-5ed3-4c32-bcc2-529349a4cfb0 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:58.330475 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8276d9a5-5ed3-4c32-bcc2-529349a4cfb0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:58.355078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:58.373088 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1298/2628] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:58.383638 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f0b8c8-a43a-427d-a6bc-581753ad359c None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:58.385501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a6f0b8c8-a43a-427d-a6bc-581753ad359c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:58.399703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8276d9a5-5ed3-4c32-bcc2-529349a4cfb0 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:58.405419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95e0550-c593-48e8-ab8a-e8e1e0d6def7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:58.406350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95e0550-c593-48e8-ab8a-e8e1e0d6def7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:58.406727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95e0550-c593-48e8-ab8a-e8e1e0d6def7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:58.407080 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95e0550-c593-48e8-ab8a-e8e1e0d6def7 admin admin] PATH_INFO: `/v3/users/dc4fe8bad42047e2827e37e8cac76a75` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:58.408359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95e0550-c593-48e8-ab8a-e8e1e0d6def7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc4fe8bad42047e2827e37e8cac76a75)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:58.411850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95e0550-c593-48e8-ab8a-e8e1e0d6def7 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:58.443628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:58.450695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f0b8c8-a43a-427d-a6bc-581753ad359c None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:58.454854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a601a1-557a-431d-8713-1e49a3722e36 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:58.455589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a601a1-557a-431d-8713-1e49a3722e36 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:58.455919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a601a1-557a-431d-8713-1e49a3722e36 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:58.456232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a601a1-557a-431d-8713-1e49a3722e36 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:58.467714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a601a1-557a-431d-8713-1e49a3722e36 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:58.468089 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a95e0550-c593-48e8-ab8a-e8e1e0d6def7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc4fe8bad42047e2827e37e8cac76a75'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.468880 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a95e0550-c593-48e8-ab8a-e8e1e0d6def7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc4fe8bad42047e2827e37e8cac76a75'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.469350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a601a1-557a-431d-8713-1e49a3722e36 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:58.476638 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a95e0550-c593-48e8-ab8a-e8e1e0d6def7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc4fe8bad42047e2827e37e8cac76a75'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.477286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a95e0550-c593-48e8-ab8a-e8e1e0d6def7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc4fe8bad42047e2827e37e8cac76a75'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.479765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1299/2629] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:58.493379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-a95e0550-c593-48e8-ab8a-e8e1e0d6def7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc4fe8bad42047e2827e37e8cac76a75'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.506828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1331/2630] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:58 2022] DELETE /identity/v3/users/dc4fe8bad42047e2827e37e8cac76a75 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:58.511966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e00a7f5-8ebe-4ea7-9743-f8f3910b90cb None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:58.512437 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e00a7f5-8ebe-4ea7-9743-f8f3910b90cb None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:58.512807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e00a7f5-8ebe-4ea7-9743-f8f3910b90cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:58.532290 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3e00a7f5-8ebe-4ea7-9743-f8f3910b90cb None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:58.533602 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3e00a7f5-8ebe-4ea7-9743-f8f3910b90cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:58.596950 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1300/2631] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:48:58.607299 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95e0550-c593-48e8-ab8a-e8e1e0d6def7 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:58.608775 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a95e0550-c593-48e8-ab8a-e8e1e0d6def7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:58.675152 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95e0550-c593-48e8-ab8a-e8e1e0d6def7 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:58.679419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425542b0-c939-49e9-92c5-c3bd6d2e4c29 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:58.680037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425542b0-c939-49e9-92c5-c3bd6d2e4c29 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:58.680475 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425542b0-c939-49e9-92c5-c3bd6d2e4c29 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:58.680756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425542b0-c939-49e9-92c5-c3bd6d2e4c29 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:58.690745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425542b0-c939-49e9-92c5-c3bd6d2e4c29 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:58.692733 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425542b0-c939-49e9-92c5-c3bd6d2e4c29 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:58.702518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1332/2632] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:48:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:48:58.797710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e00a7f5-8ebe-4ea7-9743-f8f3910b90cb None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:58.802450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3e00a7f5-8ebe-4ea7-9743-f8f3910b90cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:58.815339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e00a7f5-8ebe-4ea7-9743-f8f3910b90cb None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:58.835161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:58.836739 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:58.837176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:58.837556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] PATH_INFO: `/v3/projects/bc1ed9ade340401b89103e12238dd8fa` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:58.839249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc1ed9ade340401b89103e12238dd8fa)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:58.842230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:58.894904 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc1ed9ade340401b89103e12238dd8fa', 'request_id': 'req-873d3f11-0933-4520-aafe-1b4c81bbda7f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.901353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc1ed9ade340401b89103e12238dd8fa', 'request_id': 'req-873d3f11-0933-4520-aafe-1b4c81bbda7f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.911555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc1ed9ade340401b89103e12238dd8fa', 'request_id': 'req-873d3f11-0933-4520-aafe-1b4c81bbda7f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.925719 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] The token cache is being invalidate because project bc1ed9ade340401b89103e12238dd8fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:58.926992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.927362 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:58.929183 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1301/2633] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:58 2022] DELETE /identity/v3/projects/bc1ed9ade340401b89103e12238dd8fa => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:58.933843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425542b0-c939-49e9-92c5-c3bd6d2e4c29 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:58.943932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425542b0-c939-49e9-92c5-c3bd6d2e4c29 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:59.001372 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7d9964-747d-402b-94c9-15fe27dbb86c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:59.001990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7d9964-747d-402b-94c9-15fe27dbb86c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:59.002264 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7d9964-747d-402b-94c9-15fe27dbb86c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:59.002534 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7d9964-747d-402b-94c9-15fe27dbb86c admin admin] PATH_INFO: `/v3/users/a1cb51d8f5c945d2aaa12ec585374210` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:59.003641 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7d9964-747d-402b-94c9-15fe27dbb86c admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1cb51d8f5c945d2aaa12ec585374210)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:59.005914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7d9964-747d-402b-94c9-15fe27dbb86c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:59.038368 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:59.040661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:59.055585 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873d3f11-0933-4520-aafe-1b4c81bbda7f admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:59.061804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:59.062674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:59.063012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:59.063409 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] PATH_INFO: `/v3/projects/61dccc193bc443c5afe1ae3c2dcab74a` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:59.064829 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] RBAC: Authorizing `identity:delete_project(project_id=61dccc193bc443c5afe1ae3c2dcab74a)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:59.066788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:59.071499 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-df7d9964-747d-402b-94c9-15fe27dbb86c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1cb51d8f5c945d2aaa12ec585374210'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.072269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-df7d9964-747d-402b-94c9-15fe27dbb86c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1cb51d8f5c945d2aaa12ec585374210'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.073885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:59.080698 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-df7d9964-747d-402b-94c9-15fe27dbb86c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1cb51d8f5c945d2aaa12ec585374210'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.081420 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-df7d9964-747d-402b-94c9-15fe27dbb86c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1cb51d8f5c945d2aaa12ec585374210'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.092749 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:59.096419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-df7d9964-747d-402b-94c9-15fe27dbb86c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1cb51d8f5c945d2aaa12ec585374210'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.110421 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1333/2634] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:58 2022] DELETE /identity/v3/users/a1cb51d8f5c945d2aaa12ec585374210 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:59.135226 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '61dccc193bc443c5afe1ae3c2dcab74a', 'request_id': 'req-ecf37ef2-d5de-4951-839b-32f0c71cf095'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.140959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '61dccc193bc443c5afe1ae3c2dcab74a', 'request_id': 'req-ecf37ef2-d5de-4951-839b-32f0c71cf095'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.149602 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '61dccc193bc443c5afe1ae3c2dcab74a', 'request_id': 'req-ecf37ef2-d5de-4951-839b-32f0c71cf095'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.162181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] The token cache is being invalidate because project 61dccc193bc443c5afe1ae3c2dcab74a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:59.162475 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.163139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.164680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1302/2635] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:59 2022] DELETE /identity/v3/projects/61dccc193bc443c5afe1ae3c2dcab74a => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:48:59.168834 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7d9964-747d-402b-94c9-15fe27dbb86c admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:59.170245 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-df7d9964-747d-402b-94c9-15fe27dbb86c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:59.241331 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7d9964-747d-402b-94c9-15fe27dbb86c admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:59.303755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1851dd88-a15a-4ed3-97f5-66aed628e0f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:59.305000 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851dd88-a15a-4ed3-97f5-66aed628e0f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:59.305421 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851dd88-a15a-4ed3-97f5-66aed628e0f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:59.305844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851dd88-a15a-4ed3-97f5-66aed628e0f5 admin admin] PATH_INFO: `/v3/users/aa41c9d3e7c44642a058aa17ad51fe33` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:59.307464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1851dd88-a15a-4ed3-97f5-66aed628e0f5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aa41c9d3e7c44642a058aa17ad51fe33)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:59.309648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1851dd88-a15a-4ed3-97f5-66aed628e0f5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:59.366747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1851dd88-a15a-4ed3-97f5-66aed628e0f5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aa41c9d3e7c44642a058aa17ad51fe33'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.367863 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1851dd88-a15a-4ed3-97f5-66aed628e0f5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aa41c9d3e7c44642a058aa17ad51fe33'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.375637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1851dd88-a15a-4ed3-97f5-66aed628e0f5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aa41c9d3e7c44642a058aa17ad51fe33'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.376336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1851dd88-a15a-4ed3-97f5-66aed628e0f5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aa41c9d3e7c44642a058aa17ad51fe33'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.390687 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1851dd88-a15a-4ed3-97f5-66aed628e0f5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aa41c9d3e7c44642a058aa17ad51fe33'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.393278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:59.412634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1334/2636] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:48:59 2022] DELETE /identity/v3/users/aa41c9d3e7c44642a058aa17ad51fe33 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:59.423994 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:59.426019 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:59.446400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf37ef2-d5de-4951-839b-32f0c71cf095 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:59.528432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:59.529276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:59.529554 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:59.529885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] PATH_INFO: `/v3/projects/26150a2d9bc1446390e11492aaf2eff5` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:59.531001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] RBAC: Authorizing `identity:delete_project(project_id=26150a2d9bc1446390e11492aaf2eff5)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:59.533003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:59.598686 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26150a2d9bc1446390e11492aaf2eff5', 'request_id': 'req-92bd3310-a885-4c21-b518-fc903ab6105b'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.605415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26150a2d9bc1446390e11492aaf2eff5', 'request_id': 'req-92bd3310-a885-4c21-b518-fc903ab6105b'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.616555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26150a2d9bc1446390e11492aaf2eff5', 'request_id': 'req-92bd3310-a885-4c21-b518-fc903ab6105b'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.631107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] The token cache is being invalidate because project 26150a2d9bc1446390e11492aaf2eff5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:59.631403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.632187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.636347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1303/2637] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:59 2022] DELETE /identity/v3/projects/26150a2d9bc1446390e11492aaf2eff5 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:59.638844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a4d19e-239e-4654-bcb8-5c6043a3af01 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:59.639159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a4d19e-239e-4654-bcb8-5c6043a3af01 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:59.639434 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a4d19e-239e-4654-bcb8-5c6043a3af01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:59.668406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-12a4d19e-239e-4654-bcb8-5c6043a3af01 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:59.669701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-12a4d19e-239e-4654-bcb8-5c6043a3af01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:59.741377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1335/2638] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:59.743380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:59.744795 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:59.749909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b1edbd-8676-49c5-b340-4d4df763b269 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:59.750256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b1edbd-8676-49c5-b340-4d4df763b269 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:59.750632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b1edbd-8676-49c5-b340-4d4df763b269 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:59.759137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd3310-a885-4c21-b518-fc903ab6105b admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:59.765584 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed208434-7022-40bc-99e9-b1a0d42d5740 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:59.766263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed208434-7022-40bc-99e9-b1a0d42d5740 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:59.767535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed208434-7022-40bc-99e9-b1a0d42d5740 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:59.767966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed208434-7022-40bc-99e9-b1a0d42d5740 admin admin] PATH_INFO: `/v3/projects/c71e1834bba241f487ddc6832200e0ef` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:59.771467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed208434-7022-40bc-99e9-b1a0d42d5740 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c71e1834bba241f487ddc6832200e0ef)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:59.771983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed208434-7022-40bc-99e9-b1a0d42d5740 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:59.774904 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-87b1edbd-8676-49c5-b340-4d4df763b269 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:59.776214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-87b1edbd-8676-49c5-b340-4d4df763b269 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:59.784340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1336/2639] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:48:59.788900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b1edbd-8676-49c5-b340-4d4df763b269 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:59.797875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b1edbd-8676-49c5-b340-4d4df763b269 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:59.798171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:48:59.803886 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bb4ef4-3de5-4570-a2df-019eb104bd8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:59.804556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bb4ef4-3de5-4570-a2df-019eb104bd8e admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:59.804859 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bb4ef4-3de5-4570-a2df-019eb104bd8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:59.805162 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bb4ef4-3de5-4570-a2df-019eb104bd8e admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:59.805811 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bb4ef4-3de5-4570-a2df-019eb104bd8e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:48:59.806116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bb4ef4-3de5-4570-a2df-019eb104bd8e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:59.807743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bb4ef4-3de5-4570-a2df-019eb104bd8e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:59.819905 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1337/2640] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:48:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:48:59.824079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bb4ef4-3de5-4570-a2df-019eb104bd8e admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:59.824232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ed208434-7022-40bc-99e9-b1a0d42d5740 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c71e1834bba241f487ddc6832200e0ef', 'request_id': 'req-ed208434-7022-40bc-99e9-b1a0d42d5740'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.829621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ed208434-7022-40bc-99e9-b1a0d42d5740 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c71e1834bba241f487ddc6832200e0ef', 'request_id': 'req-ed208434-7022-40bc-99e9-b1a0d42d5740'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.834755 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bb4ef4-3de5-4570-a2df-019eb104bd8e admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:48:59.839598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ed208434-7022-40bc-99e9-b1a0d42d5740 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c71e1834bba241f487ddc6832200e0ef', 'request_id': 'req-ed208434-7022-40bc-99e9-b1a0d42d5740'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.852511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ed208434-7022-40bc-99e9-b1a0d42d5740 admin admin] The token cache is being invalidate because project c71e1834bba241f487ddc6832200e0ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:48:59.852824 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ed208434-7022-40bc-99e9-b1a0d42d5740 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.853516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-ed208434-7022-40bc-99e9-b1a0d42d5740 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:48:59.855893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1304/2641] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:48:59 2022] DELETE /identity/v3/projects/c71e1834bba241f487ddc6832200e0ef => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:48:59.861629 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ebd9b-ffbf-483e-97dd-29d824ca6963 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:59.861909 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ebd9b-ffbf-483e-97dd-29d824ca6963 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:59.862181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ebd9b-ffbf-483e-97dd-29d824ca6963 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:59.884290 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-286ebd9b-ffbf-483e-97dd-29d824ca6963 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:59.885625 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-286ebd9b-ffbf-483e-97dd-29d824ca6963 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:59.900279 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cacb0ee-fd39-4a2c-a0e3-4de953e90bb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:48:59.900913 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cacb0ee-fd39-4a2c-a0e3-4de953e90bb2 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:59.901180 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cacb0ee-fd39-4a2c-a0e3-4de953e90bb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:59.901450 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cacb0ee-fd39-4a2c-a0e3-4de953e90bb2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:59.902083 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cacb0ee-fd39-4a2c-a0e3-4de953e90bb2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:48:59.904292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cacb0ee-fd39-4a2c-a0e3-4de953e90bb2 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:48:59.939064 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1338/2642] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:48:59 2022] POST /identity/v3/projects => generated 383 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:48:59.943047 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cacb0ee-fd39-4a2c-a0e3-4de953e90bb2 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:48:59.945250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-9cacb0ee-fd39-4a2c-a0e3-4de953e90bb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:48:59.959491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1305/2643] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:48:59.967057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82caaf4-12c2-4110-90db-f29fd9520f05 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:48:59.967426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82caaf4-12c2-4110-90db-f29fd9520f05 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:48:59.967860 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82caaf4-12c2-4110-90db-f29fd9520f05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:48:59.992753 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-e82caaf4-12c2-4110-90db-f29fd9520f05 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:48:59.993955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e82caaf4-12c2-4110-90db-f29fd9520f05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:00.015060 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:INSERT=2,keystone:DELETE=1,keystone:SELECT=97 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:00.019104 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cacb0ee-fd39-4a2c-a0e3-4de953e90bb2 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:00.027407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121f4e5f-a770-44e5-b893-f617decd9c85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:00.028008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121f4e5f-a770-44e5-b893-f617decd9c85 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.028637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121f4e5f-a770-44e5-b893-f617decd9c85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.029298 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121f4e5f-a770-44e5-b893-f617decd9c85 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.030065 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121f4e5f-a770-44e5-b893-f617decd9c85 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:00.032231 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121f4e5f-a770-44e5-b893-f617decd9c85 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:00.054177 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1339/2644] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:48:59 2022] POST /identity/v3/users => generated 292 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:49:00.058166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121f4e5f-a770-44e5-b893-f617decd9c85 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:00.059122 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1306/2645] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:48:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:49:00.062955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82caaf4-12c2-4110-90db-f29fd9520f05 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:00.069460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121f4e5f-a770-44e5-b893-f617decd9c85 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:00.071671 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82caaf4-12c2-4110-90db-f29fd9520f05 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:00.074499 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e3bec7-7bf3-405e-8cfb-d874c82b6b4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:00.075194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e3bec7-7bf3-405e-8cfb-d874c82b6b4c admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.075596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e3bec7-7bf3-405e-8cfb-d874c82b6b4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.075949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e3bec7-7bf3-405e-8cfb-d874c82b6b4c admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.076171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4661cfad-1ae5-470c-baae-3703fd07264e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:00.076628 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e3bec7-7bf3-405e-8cfb-d874c82b6b4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:00.076777 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4661cfad-1ae5-470c-baae-3703fd07264e admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.077060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4661cfad-1ae5-470c-baae-3703fd07264e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.077327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4661cfad-1ae5-470c-baae-3703fd07264e admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.077959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4661cfad-1ae5-470c-baae-3703fd07264e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:49:00.078256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4661cfad-1ae5-470c-baae-3703fd07264e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:00.078354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e3bec7-7bf3-405e-8cfb-d874c82b6b4c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:00.079937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4661cfad-1ae5-470c-baae-3703fd07264e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:00.089088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1340/2646] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:00 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:49:00.093186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e3bec7-7bf3-405e-8cfb-d874c82b6b4c admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:00.093648 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1307/2647] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:49:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:49:00.097209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4661cfad-1ae5-470c-baae-3703fd07264e admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:00.104247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e3bec7-7bf3-405e-8cfb-d874c82b6b4c admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:00.108187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4661cfad-1ae5-470c-baae-3703fd07264e admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:00.109209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3573dad9-204e-4b28-a017-cad8f4947b43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:00.109800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3573dad9-204e-4b28-a017-cad8f4947b43 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.110078 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3573dad9-204e-4b28-a017-cad8f4947b43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.110364 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3573dad9-204e-4b28-a017-cad8f4947b43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.111001 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3573dad9-204e-4b28-a017-cad8f4947b43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:00.112860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3573dad9-204e-4b28-a017-cad8f4947b43 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:00.114704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71ae436-14a0-47fc-8e4b-2c187f64777d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:00.115714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71ae436-14a0-47fc-8e4b-2c187f64777d admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.116007 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71ae436-14a0-47fc-8e4b-2c187f64777d admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.116282 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71ae436-14a0-47fc-8e4b-2c187f64777d admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.116952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71ae436-14a0-47fc-8e4b-2c187f64777d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:00.118863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71ae436-14a0-47fc-8e4b-2c187f64777d admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:00.124161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1341/2648] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:00 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:00.128293 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3573dad9-204e-4b28-a017-cad8f4947b43 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:00.139408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3573dad9-204e-4b28-a017-cad8f4947b43 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:00.143962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c84301-a7fa-4a88-a07c-9f82dfcbae3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:00.144649 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c84301-a7fa-4a88-a07c-9f82dfcbae3d admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.145478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c84301-a7fa-4a88-a07c-9f82dfcbae3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.145825 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c84301-a7fa-4a88-a07c-9f82dfcbae3d admin admin] PATH_INFO: `/v3/projects/fd06e187240c44a4a61fa1349aa415da/users/67fc86542293480cb55ae1f128719466/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.160837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c84301-a7fa-4a88-a07c-9f82dfcbae3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd06e187240c44a4a61fa1349aa415da, user_id=67fc86542293480cb55ae1f128719466, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:00.166420 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1308/2649] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:49:00 2022] POST /identity/v3/projects => generated 383 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:49:00.166574 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c84301-a7fa-4a88-a07c-9f82dfcbae3d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:00.167736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71ae436-14a0-47fc-8e4b-2c187f64777d admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:00.177360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1342/2650] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:00 2022] PUT /identity/v3/projects/fd06e187240c44a4a61fa1349aa415da/users/67fc86542293480cb55ae1f128719466/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:00.187250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=92,keystone:DELETE=6,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:00.188556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2479aae3-740d-4150-8399-48e61af4e35c None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.188858 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2479aae3-740d-4150-8399-48e61af4e35c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.189189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2479aae3-740d-4150-8399-48e61af4e35c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.218969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-2479aae3-740d-4150-8399-48e61af4e35c None None] MFA Rules not processed for user `67fc86542293480cb55ae1f128719466`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:00.220198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2479aae3-740d-4150-8399-48e61af4e35c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:00.259113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71ae436-14a0-47fc-8e4b-2c187f64777d admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:00.286735 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1343/2651] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:00 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:49:00.298337 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2479aae3-740d-4150-8399-48e61af4e35c None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:00.299851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2479aae3-740d-4150-8399-48e61af4e35c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:00.317831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5306f3-0b6c-4c51-a5c5-0491e558a8b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:00.318487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5306f3-0b6c-4c51-a5c5-0491e558a8b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.318769 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5306f3-0b6c-4c51-a5c5-0491e558a8b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.319045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5306f3-0b6c-4c51-a5c5-0491e558a8b0 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.319668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5306f3-0b6c-4c51-a5c5-0491e558a8b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:00.322605 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5306f3-0b6c-4c51-a5c5-0491e558a8b0 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:00.344701 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1309/2652] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:49:00 2022] POST /identity/v3/users => generated 292 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:49:00.349314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5306f3-0b6c-4c51-a5c5-0491e558a8b0 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:00.358970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5306f3-0b6c-4c51-a5c5-0491e558a8b0 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:00.361549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2479aae3-740d-4150-8399-48e61af4e35c None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:00.364842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a022f6bb-e413-47d8-a08d-9dc7a97f5f5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:00.365454 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a022f6bb-e413-47d8-a08d-9dc7a97f5f5b admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.365735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a022f6bb-e413-47d8-a08d-9dc7a97f5f5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.365838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb97938c-cd8a-4e74-a622-a71032312024 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:00.366030 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a022f6bb-e413-47d8-a08d-9dc7a97f5f5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.366389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb97938c-cd8a-4e74-a622-a71032312024 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.366638 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a022f6bb-e413-47d8-a08d-9dc7a97f5f5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:00.366748 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb97938c-cd8a-4e74-a622-a71032312024 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.366950 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb97938c-cd8a-4e74-a622-a71032312024 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.368216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a022f6bb-e413-47d8-a08d-9dc7a97f5f5b admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:00.375418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb97938c-cd8a-4e74-a622-a71032312024 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:00.376863 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb97938c-cd8a-4e74-a622-a71032312024 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:00.381154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1310/2653] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:00 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:49:00.382172 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a022f6bb-e413-47d8-a08d-9dc7a97f5f5b admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:00.387043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1344/2654] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:00 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:49:00.392089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a022f6bb-e413-47d8-a08d-9dc7a97f5f5b admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:00.397157 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d48c901-aea0-404b-9da7-9607070d51d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:00.397746 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d48c901-aea0-404b-9da7-9607070d51d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.398144 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d48c901-aea0-404b-9da7-9607070d51d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.398426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d48c901-aea0-404b-9da7-9607070d51d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.399064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d48c901-aea0-404b-9da7-9607070d51d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:00.400789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d48c901-aea0-404b-9da7-9607070d51d8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:00.405916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb97938c-cd8a-4e74-a622-a71032312024 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:00.414550 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1311/2655] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:00.419051 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb97938c-cd8a-4e74-a622-a71032312024 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:00.419292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d48c901-aea0-404b-9da7-9607070d51d8 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:00.424724 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af981964-5970-4c03-b9a1-22d3ca1e2cd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:00.425519 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af981964-5970-4c03-b9a1-22d3ca1e2cd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.425829 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af981964-5970-4c03-b9a1-22d3ca1e2cd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.426684 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af981964-5970-4c03-b9a1-22d3ca1e2cd5 admin admin] PATH_INFO: `/v3/users/67fc86542293480cb55ae1f128719466` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.428482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af981964-5970-4c03-b9a1-22d3ca1e2cd5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=67fc86542293480cb55ae1f128719466)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:00.430377 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af981964-5970-4c03-b9a1-22d3ca1e2cd5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:00.430663 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d48c901-aea0-404b-9da7-9607070d51d8 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:00.436041 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5bbf52-c4d7-43aa-a435-b8bd83c27d71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:00.437496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5bbf52-c4d7-43aa-a435-b8bd83c27d71 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.437496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5bbf52-c4d7-43aa-a435-b8bd83c27d71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.437713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5bbf52-c4d7-43aa-a435-b8bd83c27d71 admin admin] PATH_INFO: `/v3/projects/02ec081cc3074a98b1a1da4532e3200d/users/f494330467ff4a1c83dcc59473b2f269/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.454778 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5bbf52-c4d7-43aa-a435-b8bd83c27d71 admin admin] RBAC: Authorizing `identity:create_grant(project_id=02ec081cc3074a98b1a1da4532e3200d, user_id=f494330467ff4a1c83dcc59473b2f269, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:00.459133 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5bbf52-c4d7-43aa-a435-b8bd83c27d71 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:00.461117 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=91,keystone:INSERT=4,keystone:DELETE=5 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:00.474165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1312/2656] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:00 2022] PUT /identity/v3/projects/02ec081cc3074a98b1a1da4532e3200d/users/f494330467ff4a1c83dcc59473b2f269/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:49:00.482945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-af981964-5970-4c03-b9a1-22d3ca1e2cd5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '67fc86542293480cb55ae1f128719466'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:00.483620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-af981964-5970-4c03-b9a1-22d3ca1e2cd5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '67fc86542293480cb55ae1f128719466'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:00.487427 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572879a0-0db1-4a64-b9e9-0ad84980c150 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.493092 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572879a0-0db1-4a64-b9e9-0ad84980c150 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.493092 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572879a0-0db1-4a64-b9e9-0ad84980c150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.493334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-af981964-5970-4c03-b9a1-22d3ca1e2cd5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '67fc86542293480cb55ae1f128719466'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:00.493334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-af981964-5970-4c03-b9a1-22d3ca1e2cd5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '67fc86542293480cb55ae1f128719466'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:00.507839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-af981964-5970-4c03-b9a1-22d3ca1e2cd5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '67fc86542293480cb55ae1f128719466'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:00.522979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-572879a0-0db1-4a64-b9e9-0ad84980c150 None None] MFA Rules not processed for user `f494330467ff4a1c83dcc59473b2f269`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:00.523954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1345/2657] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:49:00 2022] DELETE /identity/v3/users/67fc86542293480cb55ae1f128719466 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:49:00.524198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-572879a0-0db1-4a64-b9e9-0ad84980c150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:00.528624 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9563316-906e-4d0c-9b96-8176b7b6728d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.528911 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9563316-906e-4d0c-9b96-8176b7b6728d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.529185 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9563316-906e-4d0c-9b96-8176b7b6728d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.548301 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-e9563316-906e-4d0c-9b96-8176b7b6728d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:00.549486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e9563316-906e-4d0c-9b96-8176b7b6728d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:00.586505 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:00.593183 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1313/2658] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:00 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:00.604132 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572879a0-0db1-4a64-b9e9-0ad84980c150 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:00.605535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-572879a0-0db1-4a64-b9e9-0ad84980c150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:00.614460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1346/2659] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:49:00.624555 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9563316-906e-4d0c-9b96-8176b7b6728d None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:00.625985 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e9563316-906e-4d0c-9b96-8176b7b6728d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:00.675499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572879a0-0db1-4a64-b9e9-0ad84980c150 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:00.681783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a9d964-4578-485f-8b2f-42359c5d4484 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:00.682398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a9d964-4578-485f-8b2f-42359c5d4484 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.682680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a9d964-4578-485f-8b2f-42359c5d4484 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.682951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a9d964-4578-485f-8b2f-42359c5d4484 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.692924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9563316-906e-4d0c-9b96-8176b7b6728d None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:00.693073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a9d964-4578-485f-8b2f-42359c5d4484 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:00.694459 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a9d964-4578-485f-8b2f-42359c5d4484 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:00.696722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac99ccc2-4a8e-46e2-96aa-7572b2a26690 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:00.697322 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac99ccc2-4a8e-46e2-96aa-7572b2a26690 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:00.697602 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac99ccc2-4a8e-46e2-96aa-7572b2a26690 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:00.697877 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac99ccc2-4a8e-46e2-96aa-7572b2a26690 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:00.704206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1314/2660] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:00 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:49:00.716916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac99ccc2-4a8e-46e2-96aa-7572b2a26690 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:00.716916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac99ccc2-4a8e-46e2-96aa-7572b2a26690 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:00.720967 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1347/2661] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:49:01.028689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a9d964-4578-485f-8b2f-42359c5d4484 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:01.030392 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f2a9d964-4578-485f-8b2f-42359c5d4484 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:01.054754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a9d964-4578-485f-8b2f-42359c5d4484 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:01.064153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6473d28d-7c2c-4f8e-a348-22259c64d364 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:01.064878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6473d28d-7c2c-4f8e-a348-22259c64d364 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:01.065113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6473d28d-7c2c-4f8e-a348-22259c64d364 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:01.065351 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6473d28d-7c2c-4f8e-a348-22259c64d364 admin admin] PATH_INFO: `/v3/projects/fd06e187240c44a4a61fa1349aa415da` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:01.068879 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6473d28d-7c2c-4f8e-a348-22259c64d364 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd06e187240c44a4a61fa1349aa415da)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:01.073449 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6473d28d-7c2c-4f8e-a348-22259c64d364 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:01.104885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac99ccc2-4a8e-46e2-96aa-7572b2a26690 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:01.107033 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ac99ccc2-4a8e-46e2-96aa-7572b2a26690 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:01.124520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac99ccc2-4a8e-46e2-96aa-7572b2a26690 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:01.126498 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a452af-71b7-48d3-a098-8f2ac8c7df68 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:01.127765 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a452af-71b7-48d3-a098-8f2ac8c7df68 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:01.128193 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a452af-71b7-48d3-a098-8f2ac8c7df68 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:01.128617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a452af-71b7-48d3-a098-8f2ac8c7df68 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:01.131286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f9a452af-71b7-48d3-a098-8f2ac8c7df68 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:01.148071 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a452af-71b7-48d3-a098-8f2ac8c7df68 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:01.149845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a452af-71b7-48d3-a098-8f2ac8c7df68 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:01.157688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6473d28d-7c2c-4f8e-a348-22259c64d364 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd06e187240c44a4a61fa1349aa415da', 'request_id': 'req-6473d28d-7c2c-4f8e-a348-22259c64d364'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:01.161151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1348/2662] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:49:01.185377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6473d28d-7c2c-4f8e-a348-22259c64d364 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd06e187240c44a4a61fa1349aa415da', 'request_id': 'req-6473d28d-7c2c-4f8e-a348-22259c64d364'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:01.197452 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6473d28d-7c2c-4f8e-a348-22259c64d364 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd06e187240c44a4a61fa1349aa415da', 'request_id': 'req-6473d28d-7c2c-4f8e-a348-22259c64d364'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:01.218094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6473d28d-7c2c-4f8e-a348-22259c64d364 admin admin] The token cache is being invalidate because project fd06e187240c44a4a61fa1349aa415da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:49:01.218683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6473d28d-7c2c-4f8e-a348-22259c64d364 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:01.220776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6473d28d-7c2c-4f8e-a348-22259c64d364 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:01.221785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1315/2663] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:49:01 2022] DELETE /identity/v3/projects/fd06e187240c44a4a61fa1349aa415da => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:01.226783 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5227c1-f65d-48f1-8296-fc376e5ee26c None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:01.227172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5227c1-f65d-48f1-8296-fc376e5ee26c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:01.227658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5227c1-f65d-48f1-8296-fc376e5ee26c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:01.250597 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4f5227c1-f65d-48f1-8296-fc376e5ee26c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:01.251797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4f5227c1-f65d-48f1-8296-fc376e5ee26c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:01.282898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:01.315801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1349/2664] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:49:01.324427 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89442f71-d21b-46ee-8039-882acdc87d24 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:01.324770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89442f71-d21b-46ee-8039-882acdc87d24 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:01.325053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89442f71-d21b-46ee-8039-882acdc87d24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:01.345311 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-89442f71-d21b-46ee-8039-882acdc87d24 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:01.346621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-89442f71-d21b-46ee-8039-882acdc87d24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:01.355410 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1316/2665] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:01.360205 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5227c1-f65d-48f1-8296-fc376e5ee26c None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:01.370659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5227c1-f65d-48f1-8296-fc376e5ee26c None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:01.375398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1360231a-30e4-4faa-bde6-b41d3baf8385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:01.376060 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1360231a-30e4-4faa-bde6-b41d3baf8385 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:01.376436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1360231a-30e4-4faa-bde6-b41d3baf8385 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:01.376776 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1360231a-30e4-4faa-bde6-b41d3baf8385 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:01.377511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1360231a-30e4-4faa-bde6-b41d3baf8385 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:49:01.377881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1360231a-30e4-4faa-bde6-b41d3baf8385 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:01.380149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1360231a-30e4-4faa-bde6-b41d3baf8385 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:01.391602 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1350/2666] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:49:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:49:01.396068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89442f71-d21b-46ee-8039-882acdc87d24 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:01.405353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89442f71-d21b-46ee-8039-882acdc87d24 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:01.410745 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa16b6e5-7d7b-41df-bd8e-1c3657176853 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:01.411399 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa16b6e5-7d7b-41df-bd8e-1c3657176853 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:01.411683 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa16b6e5-7d7b-41df-bd8e-1c3657176853 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:01.411948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa16b6e5-7d7b-41df-bd8e-1c3657176853 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:01.412621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa16b6e5-7d7b-41df-bd8e-1c3657176853 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:01.414561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa16b6e5-7d7b-41df-bd8e-1c3657176853 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:01.450917 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1317/2667] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:49:01 2022] POST /identity/v3/projects => generated 371 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:49:01.452118 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1360231a-30e4-4faa-bde6-b41d3baf8385 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:01.462240 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1360231a-30e4-4faa-bde6-b41d3baf8385 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:01.523672 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d501fdcf-c1f0-4445-9fa0-cdde9413ea53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:01.524419 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d501fdcf-c1f0-4445-9fa0-cdde9413ea53 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:01.524703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d501fdcf-c1f0-4445-9fa0-cdde9413ea53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:01.524976 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d501fdcf-c1f0-4445-9fa0-cdde9413ea53 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:01.525638 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d501fdcf-c1f0-4445-9fa0-cdde9413ea53 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:01.527696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d501fdcf-c1f0-4445-9fa0-cdde9413ea53 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:01.554530 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1351/2668] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:49:01 2022] POST /identity/v3/users => generated 286 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:49:01.555793 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa16b6e5-7d7b-41df-bd8e-1c3657176853 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:01.570694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa16b6e5-7d7b-41df-bd8e-1c3657176853 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:01.577062 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d276cc2e-3a2d-4d86-9671-1c5d8f0c2627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:01.577661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d276cc2e-3a2d-4d86-9671-1c5d8f0c2627 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:01.577941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d276cc2e-3a2d-4d86-9671-1c5d8f0c2627 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:01.578209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d276cc2e-3a2d-4d86-9671-1c5d8f0c2627 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:01.578850 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d276cc2e-3a2d-4d86-9671-1c5d8f0c2627 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:01.580526 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d276cc2e-3a2d-4d86-9671-1c5d8f0c2627 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:01.595889 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1318/2669] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:01 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:01.597340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d501fdcf-c1f0-4445-9fa0-cdde9413ea53 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:01.608090 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d501fdcf-c1f0-4445-9fa0-cdde9413ea53 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:01.612978 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8a3182-059e-466c-b7cc-054b22ed6feb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:01.614266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8a3182-059e-466c-b7cc-054b22ed6feb admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:01.614541 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8a3182-059e-466c-b7cc-054b22ed6feb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:01.614813 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8a3182-059e-466c-b7cc-054b22ed6feb admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:01.615519 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8a3182-059e-466c-b7cc-054b22ed6feb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:01.617148 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8a3182-059e-466c-b7cc-054b22ed6feb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:01.628727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1352/2670] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:01 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:01.632964 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d276cc2e-3a2d-4d86-9671-1c5d8f0c2627 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:01.643766 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d276cc2e-3a2d-4d86-9671-1c5d8f0c2627 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:01.648712 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77c11d0-8145-4be5-a190-4cf7bc0a183c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:01.649360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77c11d0-8145-4be5-a190-4cf7bc0a183c admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:01.649636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77c11d0-8145-4be5-a190-4cf7bc0a183c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:01.649907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77c11d0-8145-4be5-a190-4cf7bc0a183c admin admin] PATH_INFO: `/v3/projects/03576185db7b431d9e5d5cd25bb5235b/users/693b2af1eda24bc3bb641b5724b7acac/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:01.667294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77c11d0-8145-4be5-a190-4cf7bc0a183c admin admin] RBAC: Authorizing `identity:create_grant(project_id=03576185db7b431d9e5d5cd25bb5235b, user_id=693b2af1eda24bc3bb641b5724b7acac, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:01.669975 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77c11d0-8145-4be5-a190-4cf7bc0a183c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:01.683929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1319/2671] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:01 2022] PUT /identity/v3/projects/03576185db7b431d9e5d5cd25bb5235b/users/693b2af1eda24bc3bb641b5724b7acac/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:01.695024 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e5322b-0a63-4606-b16d-c34a60f55127 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:01.695313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e5322b-0a63-4606-b16d-c34a60f55127 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:01.695608 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e5322b-0a63-4606-b16d-c34a60f55127 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:01.727086 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77c11d0-8145-4be5-a190-4cf7bc0a183c admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:01.729256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a77c11d0-8145-4be5-a190-4cf7bc0a183c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:01.730509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-d1e5322b-0a63-4606-b16d-c34a60f55127 None None] MFA Rules not processed for user `693b2af1eda24bc3bb641b5724b7acac`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:01.731756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d1e5322b-0a63-4606-b16d-c34a60f55127 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:01.757307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:01.799508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1353/2672] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:01 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:01.803160 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77c11d0-8145-4be5-a190-4cf7bc0a183c admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:01.807979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6743e399-9d07-443d-b889-7aaa0310e1ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:01.808636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6743e399-9d07-443d-b889-7aaa0310e1ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:01.808940 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6743e399-9d07-443d-b889-7aaa0310e1ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:01.809213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6743e399-9d07-443d-b889-7aaa0310e1ae admin admin] PATH_INFO: `/v3/users/7e428194fecb425fb4e1264ce6c49051` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:01.810356 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6743e399-9d07-443d-b889-7aaa0310e1ae admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e428194fecb425fb4e1264ce6c49051)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:01.812277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6743e399-9d07-443d-b889-7aaa0310e1ae admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:01.812812 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e5322b-0a63-4606-b16d-c34a60f55127 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:01.814170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d1e5322b-0a63-4606-b16d-c34a60f55127 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:01.851017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:01.861443 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6743e399-9d07-443d-b889-7aaa0310e1ae admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e428194fecb425fb4e1264ce6c49051'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:01.862138 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6743e399-9d07-443d-b889-7aaa0310e1ae admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e428194fecb425fb4e1264ce6c49051'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:01.862759 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6743e399-9d07-443d-b889-7aaa0310e1ae admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e428194fecb425fb4e1264ce6c49051'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:01.870800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6743e399-9d07-443d-b889-7aaa0310e1ae admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e428194fecb425fb4e1264ce6c49051'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:01.876256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e5322b-0a63-4606-b16d-c34a60f55127 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:01.879907 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6640be4-3927-4bc4-aba4-88639b748678 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '815504693bcc40b6aef06aacb2a9a35b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:01.880537 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6640be4-3927-4bc4-aba4-88639b748678 service glance] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:01.880817 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6640be4-3927-4bc4-aba4-88639b748678 service glance] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:01.881145 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6640be4-3927-4bc4-aba4-88639b748678 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:01.882539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-6743e399-9d07-443d-b889-7aaa0310e1ae admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e428194fecb425fb4e1264ce6c49051'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:01.882966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e6640be4-3927-4bc4-aba4-88639b748678 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:01.894159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6640be4-3927-4bc4-aba4-88639b748678 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:01.895634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6640be4-3927-4bc4-aba4-88639b748678 service glance] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:01.895926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1320/2673] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:49:01 2022] DELETE /identity/v3/users/7e428194fecb425fb4e1264ce6c49051 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:49:01.904933 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1354/2674] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:01 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:49:02.092270 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6743e399-9d07-443d-b889-7aaa0310e1ae admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:02.095416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6743e399-9d07-443d-b889-7aaa0310e1ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:02.178089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6743e399-9d07-443d-b889-7aaa0310e1ae admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:02.185068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3b919f-8936-48f5-aba4-ef5783540e8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:02.185748 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3b919f-8936-48f5-aba4-ef5783540e8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:02.186029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3b919f-8936-48f5-aba4-ef5783540e8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:02.186307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3b919f-8936-48f5-aba4-ef5783540e8e admin admin] PATH_INFO: `/v3/projects/9fedaf31a9b64cbfb658837f17204f66` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:02.189277 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3b919f-8936-48f5-aba4-ef5783540e8e admin admin] RBAC: Authorizing `identity:delete_project(project_id=9fedaf31a9b64cbfb658837f17204f66)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:02.190691 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3b919f-8936-48f5-aba4-ef5783540e8e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:02.241790 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:02.251049 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-4a3b919f-8936-48f5-aba4-ef5783540e8e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9fedaf31a9b64cbfb658837f17204f66', 'request_id': 'req-4a3b919f-8936-48f5-aba4-ef5783540e8e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:02.260023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-4a3b919f-8936-48f5-aba4-ef5783540e8e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9fedaf31a9b64cbfb658837f17204f66', 'request_id': 'req-4a3b919f-8936-48f5-aba4-ef5783540e8e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:02.278842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-4a3b919f-8936-48f5-aba4-ef5783540e8e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9fedaf31a9b64cbfb658837f17204f66', 'request_id': 'req-4a3b919f-8936-48f5-aba4-ef5783540e8e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:02.298367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-4a3b919f-8936-48f5-aba4-ef5783540e8e admin admin] The token cache is being invalidate because project 9fedaf31a9b64cbfb658837f17204f66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:49:02.298809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-4a3b919f-8936-48f5-aba4-ef5783540e8e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:02.299469 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-4a3b919f-8936-48f5-aba4-ef5783540e8e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:02.301842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1321/2675] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:49:02 2022] DELETE /identity/v3/projects/9fedaf31a9b64cbfb658837f17204f66 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:02.307973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173a171b-9038-4dd9-bd71-0af936c13fac None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:02.308472 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173a171b-9038-4dd9-bd71-0af936c13fac None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:02.308573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173a171b-9038-4dd9-bd71-0af936c13fac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:02.343871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-173a171b-9038-4dd9-bd71-0af936c13fac None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:02.346654 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-173a171b-9038-4dd9-bd71-0af936c13fac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:02.468745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1355/2676] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:02.477448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c43a01-1eca-45af-9863-ca71b1467b6f None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:02.477735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c43a01-1eca-45af-9863-ca71b1467b6f None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:02.478077 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c43a01-1eca-45af-9863-ca71b1467b6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:02.516555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-e4c43a01-1eca-45af-9863-ca71b1467b6f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:02.517854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e4c43a01-1eca-45af-9863-ca71b1467b6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:02.532297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1322/2677] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:02.537163 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173a171b-9038-4dd9-bd71-0af936c13fac None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:02.547343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173a171b-9038-4dd9-bd71-0af936c13fac None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:02.552579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae72376-86d3-493f-aead-7fa333324370 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:02.553253 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae72376-86d3-493f-aead-7fa333324370 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:02.553600 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae72376-86d3-493f-aead-7fa333324370 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:02.553942 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae72376-86d3-493f-aead-7fa333324370 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:02.554749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae72376-86d3-493f-aead-7fa333324370 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:49:02.555134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae72376-86d3-493f-aead-7fa333324370 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:02.556877 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae72376-86d3-493f-aead-7fa333324370 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:02.570362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1356/2678] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:49:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:49:02.574616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c43a01-1eca-45af-9863-ca71b1467b6f None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:02.585572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c43a01-1eca-45af-9863-ca71b1467b6f None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:02.590281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca40ca7-ad0d-4dd9-9283-c3159a3ffcb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:02.590870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca40ca7-ad0d-4dd9-9283-c3159a3ffcb4 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:02.591182 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca40ca7-ad0d-4dd9-9283-c3159a3ffcb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:02.591450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca40ca7-ad0d-4dd9-9283-c3159a3ffcb4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:02.592084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca40ca7-ad0d-4dd9-9283-c3159a3ffcb4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:02.593998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca40ca7-ad0d-4dd9-9283-c3159a3ffcb4 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:02.626760 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1323/2679] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:49:02 2022] POST /identity/v3/projects => generated 373 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:49:02.627771 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae72376-86d3-493f-aead-7fa333324370 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:02.638200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae72376-86d3-493f-aead-7fa333324370 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:02.694796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f872b516-13d1-4cac-8797-1d56f41abb94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:02.695392 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f872b516-13d1-4cac-8797-1d56f41abb94 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:02.695669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f872b516-13d1-4cac-8797-1d56f41abb94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:02.696037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f872b516-13d1-4cac-8797-1d56f41abb94 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:02.696881 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f872b516-13d1-4cac-8797-1d56f41abb94 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:02.698739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f872b516-13d1-4cac-8797-1d56f41abb94 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:02.722375 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1357/2680] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:49:02 2022] POST /identity/v3/users => generated 287 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:49:02.727296 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca40ca7-ad0d-4dd9-9283-c3159a3ffcb4 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:02.738433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca40ca7-ad0d-4dd9-9283-c3159a3ffcb4 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:02.743058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a06c74-4055-4953-9c42-b0b250dc5975 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:02.743643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a06c74-4055-4953-9c42-b0b250dc5975 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:02.743919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a06c74-4055-4953-9c42-b0b250dc5975 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:02.744186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a06c74-4055-4953-9c42-b0b250dc5975 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:02.744851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a06c74-4055-4953-9c42-b0b250dc5975 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:02.746435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a06c74-4055-4953-9c42-b0b250dc5975 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:02.756149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1324/2681] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:02 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:49:02.760006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f872b516-13d1-4cac-8797-1d56f41abb94 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:02.770287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f872b516-13d1-4cac-8797-1d56f41abb94 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:02.774780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabbc41b-9e46-4e62-98ff-fbd0193aa221 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:02.775360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabbc41b-9e46-4e62-98ff-fbd0193aa221 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:02.775747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabbc41b-9e46-4e62-98ff-fbd0193aa221 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:02.776077 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabbc41b-9e46-4e62-98ff-fbd0193aa221 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:02.776842 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabbc41b-9e46-4e62-98ff-fbd0193aa221 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:02.778402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabbc41b-9e46-4e62-98ff-fbd0193aa221 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:02.790842 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1358/2682] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:02.794395 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a06c74-4055-4953-9c42-b0b250dc5975 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:02.804632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a06c74-4055-4953-9c42-b0b250dc5975 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:02.809464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6846feea-ad99-4dbd-9f4d-74ed44ba3169 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:02.810102 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6846feea-ad99-4dbd-9f4d-74ed44ba3169 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:02.810379 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6846feea-ad99-4dbd-9f4d-74ed44ba3169 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:02.810652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6846feea-ad99-4dbd-9f4d-74ed44ba3169 admin admin] PATH_INFO: `/v3/projects/325a146c46e54eec898106e7b88a86b6/users/b287a26201c142f983d844c07df169e9/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:02.826919 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6846feea-ad99-4dbd-9f4d-74ed44ba3169 admin admin] RBAC: Authorizing `identity:create_grant(project_id=325a146c46e54eec898106e7b88a86b6, user_id=b287a26201c142f983d844c07df169e9, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:02.829816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6846feea-ad99-4dbd-9f4d-74ed44ba3169 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:02.844385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1325/2683] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:02 2022] PUT /identity/v3/projects/325a146c46e54eec898106e7b88a86b6/users/b287a26201c142f983d844c07df169e9/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:49:02.850095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbd6181-58db-41bb-b7bc-0d10bfae0dcb None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:02.850385 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbd6181-58db-41bb-b7bc-0d10bfae0dcb None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:02.850673 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbd6181-58db-41bb-b7bc-0d10bfae0dcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:02.870526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-2fbd6181-58db-41bb-b7bc-0d10bfae0dcb None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:02.871739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2fbd6181-58db-41bb-b7bc-0d10bfae0dcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:02.905073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:02.933212 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1359/2684] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:02.942840 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6846feea-ad99-4dbd-9f4d-74ed44ba3169 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:02.944561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6846feea-ad99-4dbd-9f4d-74ed44ba3169 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:03.010365 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6846feea-ad99-4dbd-9f4d-74ed44ba3169 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:03.014206 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907489e0-ebfc-4d76-8df5-57f5a2d7fab9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:03.014810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907489e0-ebfc-4d76-8df5-57f5a2d7fab9 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:03.015113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907489e0-ebfc-4d76-8df5-57f5a2d7fab9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:03.015408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907489e0-ebfc-4d76-8df5-57f5a2d7fab9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:03.026723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907489e0-ebfc-4d76-8df5-57f5a2d7fab9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:03.028203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907489e0-ebfc-4d76-8df5-57f5a2d7fab9 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:03.038789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1326/2685] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:49:03.626116 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbd6181-58db-41bb-b7bc-0d10bfae0dcb None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:03.627827 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2fbd6181-58db-41bb-b7bc-0d10bfae0dcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:03.645344 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbd6181-58db-41bb-b7bc-0d10bfae0dcb None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:03.655238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8213dcb5-8768-43c0-b7e0-1b7651fbd54d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:03.655888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8213dcb5-8768-43c0-b7e0-1b7651fbd54d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:03.656288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8213dcb5-8768-43c0-b7e0-1b7651fbd54d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:03.656665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8213dcb5-8768-43c0-b7e0-1b7651fbd54d admin admin] PATH_INFO: `/v3/users/f494330467ff4a1c83dcc59473b2f269` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:03.658521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8213dcb5-8768-43c0-b7e0-1b7651fbd54d admin admin] RBAC: Authorizing `identity:delete_user(user_id=f494330467ff4a1c83dcc59473b2f269)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:03.660789 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8213dcb5-8768-43c0-b7e0-1b7651fbd54d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:03.745851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8213dcb5-8768-43c0-b7e0-1b7651fbd54d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f494330467ff4a1c83dcc59473b2f269'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:03.746732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8213dcb5-8768-43c0-b7e0-1b7651fbd54d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f494330467ff4a1c83dcc59473b2f269'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:03.762583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8213dcb5-8768-43c0-b7e0-1b7651fbd54d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f494330467ff4a1c83dcc59473b2f269'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:03.762930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8213dcb5-8768-43c0-b7e0-1b7651fbd54d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f494330467ff4a1c83dcc59473b2f269'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:03.783464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-8213dcb5-8768-43c0-b7e0-1b7651fbd54d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f494330467ff4a1c83dcc59473b2f269'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:03.813599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9993a3f-733e-421d-a2bf-52eff0298ccf None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:03.814055 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9993a3f-733e-421d-a2bf-52eff0298ccf None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:03.814156 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9993a3f-733e-421d-a2bf-52eff0298ccf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:03.814345 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1360/2686] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:49:03 2022] DELETE /identity/v3/users/f494330467ff4a1c83dcc59473b2f269 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:03.854052 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-c9993a3f-733e-421d-a2bf-52eff0298ccf None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:03.854978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c9993a3f-733e-421d-a2bf-52eff0298ccf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:03.996815 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1327/2687] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:04.015219 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8213dcb5-8768-43c0-b7e0-1b7651fbd54d admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:04.017164 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8213dcb5-8768-43c0-b7e0-1b7651fbd54d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:04.108920 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8213dcb5-8768-43c0-b7e0-1b7651fbd54d admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:04.116944 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b575d68c-f0a5-48ce-a976-13f9fa41454f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:04.117918 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b575d68c-f0a5-48ce-a976-13f9fa41454f service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:04.122088 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b575d68c-f0a5-48ce-a976-13f9fa41454f service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:04.122544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b575d68c-f0a5-48ce-a976-13f9fa41454f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:04.150023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b575d68c-f0a5-48ce-a976-13f9fa41454f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:04.151497 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b575d68c-f0a5-48ce-a976-13f9fa41454f service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:04.165589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1361/2688] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:49:04.637359 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9993a3f-733e-421d-a2bf-52eff0298ccf None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:04.642291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c9993a3f-733e-421d-a2bf-52eff0298ccf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:04.669604 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9993a3f-733e-421d-a2bf-52eff0298ccf None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:04.682813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:04.683488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:04.683772 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:04.684050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] PATH_INFO: `/v3/projects/02ec081cc3074a98b1a1da4532e3200d` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:04.685355 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] RBAC: Authorizing `identity:delete_project(project_id=02ec081cc3074a98b1a1da4532e3200d)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:04.687618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:04.692729 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:04.774256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02ec081cc3074a98b1a1da4532e3200d', 'request_id': 'req-cd8b199e-f6df-41cc-9c5d-996c7755470d'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:04.780006 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02ec081cc3074a98b1a1da4532e3200d', 'request_id': 'req-cd8b199e-f6df-41cc-9c5d-996c7755470d'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:04.790327 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02ec081cc3074a98b1a1da4532e3200d', 'request_id': 'req-cd8b199e-f6df-41cc-9c5d-996c7755470d'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:04.803462 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] The token cache is being invalidate because project 02ec081cc3074a98b1a1da4532e3200d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:49:04.803857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:04.804621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:04.806853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1328/2689] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:49:04 2022] DELETE /identity/v3/projects/02ec081cc3074a98b1a1da4532e3200d => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:07.056394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b575d68c-f0a5-48ce-a976-13f9fa41454f service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:07.058877 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b575d68c-f0a5-48ce-a976-13f9fa41454f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:07.079379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b575d68c-f0a5-48ce-a976-13f9fa41454f service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:07.154139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:07.170956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfe15b8-d0f0-4724-981b-2c3bca06bcf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:07.171876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfe15b8-d0f0-4724-981b-2c3bca06bcf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:07.176872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfe15b8-d0f0-4724-981b-2c3bca06bcf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:07.177298 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfe15b8-d0f0-4724-981b-2c3bca06bcf3 admin admin] PATH_INFO: `/v3/users/0d9d9929db5f4150a12b29b0b39a8eaf` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:07.178799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfe15b8-d0f0-4724-981b-2c3bca06bcf3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d9d9929db5f4150a12b29b0b39a8eaf)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:07.180921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfe15b8-d0f0-4724-981b-2c3bca06bcf3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:07.269680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4bfe15b8-d0f0-4724-981b-2c3bca06bcf3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d9d9929db5f4150a12b29b0b39a8eaf'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:07.274318 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4bfe15b8-d0f0-4724-981b-2c3bca06bcf3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d9d9929db5f4150a12b29b0b39a8eaf'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:07.283470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4bfe15b8-d0f0-4724-981b-2c3bca06bcf3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d9d9929db5f4150a12b29b0b39a8eaf'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:07.284491 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4bfe15b8-d0f0-4724-981b-2c3bca06bcf3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d9d9929db5f4150a12b29b0b39a8eaf'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:07.312867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-4bfe15b8-d0f0-4724-981b-2c3bca06bcf3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d9d9929db5f4150a12b29b0b39a8eaf'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:07.336354 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1362/2690] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:49:07 2022] DELETE /identity/v3/users/0d9d9929db5f4150a12b29b0b39a8eaf => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:07.565545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:07.567858 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:07.584931 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8b199e-f6df-41cc-9c5d-996c7755470d admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:07.720201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ee3ff7-c5b2-458a-815b-667af2d9bb61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:07.720927 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ee3ff7-c5b2-458a-815b-667af2d9bb61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:07.721199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ee3ff7-c5b2-458a-815b-667af2d9bb61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:07.721471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ee3ff7-c5b2-458a-815b-667af2d9bb61 admin admin] PATH_INFO: `/v3/projects/5b99b160db0f4b24a875642de028f2f8` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:07.723565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ee3ff7-c5b2-458a-815b-667af2d9bb61 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b99b160db0f4b24a875642de028f2f8)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:07.727219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ee3ff7-c5b2-458a-815b-667af2d9bb61 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:07.805582 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-17ee3ff7-c5b2-458a-815b-667af2d9bb61 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b99b160db0f4b24a875642de028f2f8', 'request_id': 'req-17ee3ff7-c5b2-458a-815b-667af2d9bb61'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:07.811822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-17ee3ff7-c5b2-458a-815b-667af2d9bb61 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b99b160db0f4b24a875642de028f2f8', 'request_id': 'req-17ee3ff7-c5b2-458a-815b-667af2d9bb61'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:07.826707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-17ee3ff7-c5b2-458a-815b-667af2d9bb61 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b99b160db0f4b24a875642de028f2f8', 'request_id': 'req-17ee3ff7-c5b2-458a-815b-667af2d9bb61'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:07.856152 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-17ee3ff7-c5b2-458a-815b-667af2d9bb61 admin admin] The token cache is being invalidate because project 5b99b160db0f4b24a875642de028f2f8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:49:07.856496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-17ee3ff7-c5b2-458a-815b-667af2d9bb61 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:07.866817 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-17ee3ff7-c5b2-458a-815b-667af2d9bb61 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:07.872620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1329/2691] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:49:07 2022] DELETE /identity/v3/projects/5b99b160db0f4b24a875642de028f2f8 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:07.876421 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfe15b8-d0f0-4724-981b-2c3bca06bcf3 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:07.880593 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4bfe15b8-d0f0-4724-981b-2c3bca06bcf3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:07.896609 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfe15b8-d0f0-4724-981b-2c3bca06bcf3 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:07.998011 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e821dc6-e874-47ae-a736-a0e96f67767c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:07.999103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e821dc6-e874-47ae-a736-a0e96f67767c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:07.999531 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e821dc6-e874-47ae-a736-a0e96f67767c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:08.000731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e821dc6-e874-47ae-a736-a0e96f67767c admin admin] PATH_INFO: `/v3/users/18d2841c2aa241abbaad07612ce5ea03` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:08.002054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e821dc6-e874-47ae-a736-a0e96f67767c admin admin] RBAC: Authorizing `identity:delete_user(user_id=18d2841c2aa241abbaad07612ce5ea03)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:08.005568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e821dc6-e874-47ae-a736-a0e96f67767c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:08.064988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1e821dc6-e874-47ae-a736-a0e96f67767c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18d2841c2aa241abbaad07612ce5ea03'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:08.065614 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1e821dc6-e874-47ae-a736-a0e96f67767c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18d2841c2aa241abbaad07612ce5ea03'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:08.074200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1e821dc6-e874-47ae-a736-a0e96f67767c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18d2841c2aa241abbaad07612ce5ea03'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:08.074934 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1e821dc6-e874-47ae-a736-a0e96f67767c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18d2841c2aa241abbaad07612ce5ea03'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:08.096463 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1e821dc6-e874-47ae-a736-a0e96f67767c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18d2841c2aa241abbaad07612ce5ea03'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:08.097413 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0248782b-fe5a-42c3-8505-a9361f41e211 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:08.097731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0248782b-fe5a-42c3-8505-a9361f41e211 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:08.098009 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0248782b-fe5a-42c3-8505-a9361f41e211 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:08.102953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:08.124558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1363/2692] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:49:07 2022] DELETE /identity/v3/users/18d2841c2aa241abbaad07612ce5ea03 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:08.134377 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-0248782b-fe5a-42c3-8505-a9361f41e211 None None] MFA Rules not processed for user `b287a26201c142f983d844c07df169e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:08.135617 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0248782b-fe5a-42c3-8505-a9361f41e211 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:08.146876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:08.223789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1330/2693] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:08 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:49:08.228195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e821dc6-e874-47ae-a736-a0e96f67767c admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:08.231893 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1e821dc6-e874-47ae-a736-a0e96f67767c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:08.343045 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e821dc6-e874-47ae-a736-a0e96f67767c admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:08.356977 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d82c93d-da67-47c4-9749-53bf2b0790f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:08.358236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d82c93d-da67-47c4-9749-53bf2b0790f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:08.358803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d82c93d-da67-47c4-9749-53bf2b0790f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:08.359589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0248782b-fe5a-42c3-8505-a9361f41e211 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:08.360216 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d82c93d-da67-47c4-9749-53bf2b0790f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:08.361611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0248782b-fe5a-42c3-8505-a9361f41e211 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:08.362010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d82c93d-da67-47c4-9749-53bf2b0790f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:08.366350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d82c93d-da67-47c4-9749-53bf2b0790f8 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:08.379293 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0248782b-fe5a-42c3-8505-a9361f41e211 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:08.390020 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234de071-7fc0-4f7b-af53-f6f0a3de21ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:08.390979 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234de071-7fc0-4f7b-af53-f6f0a3de21ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:08.391262 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234de071-7fc0-4f7b-af53-f6f0a3de21ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:08.391535 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234de071-7fc0-4f7b-af53-f6f0a3de21ee admin admin] PATH_INFO: `/v3/projects/ae9d113dac8742fc9e957c2b7f6ce7c5` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:08.392777 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234de071-7fc0-4f7b-af53-f6f0a3de21ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae9d113dac8742fc9e957c2b7f6ce7c5)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:08.402110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234de071-7fc0-4f7b-af53-f6f0a3de21ee admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:08.414641 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1364/2694] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:49:08 2022] POST /identity/v3/projects => generated 373 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:49:08.418568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d82c93d-da67-47c4-9749-53bf2b0790f8 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:08.475122 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-234de071-7fc0-4f7b-af53-f6f0a3de21ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae9d113dac8742fc9e957c2b7f6ce7c5', 'request_id': 'req-234de071-7fc0-4f7b-af53-f6f0a3de21ee'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:08.481842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-234de071-7fc0-4f7b-af53-f6f0a3de21ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae9d113dac8742fc9e957c2b7f6ce7c5', 'request_id': 'req-234de071-7fc0-4f7b-af53-f6f0a3de21ee'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:08.498342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-234de071-7fc0-4f7b-af53-f6f0a3de21ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae9d113dac8742fc9e957c2b7f6ce7c5', 'request_id': 'req-234de071-7fc0-4f7b-af53-f6f0a3de21ee'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:08.516780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-234de071-7fc0-4f7b-af53-f6f0a3de21ee admin admin] The token cache is being invalidate because project ae9d113dac8742fc9e957c2b7f6ce7c5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:49:08.517446 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-234de071-7fc0-4f7b-af53-f6f0a3de21ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:08.518436 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-234de071-7fc0-4f7b-af53-f6f0a3de21ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:08.524563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1331/2695] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:49:08 2022] DELETE /identity/v3/projects/ae9d113dac8742fc9e957c2b7f6ce7c5 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:08.526551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f2845b-1cac-4a0d-933e-dea82d7aacee None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:08.526837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f2845b-1cac-4a0d-933e-dea82d7aacee None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:08.527111 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f2845b-1cac-4a0d-933e-dea82d7aacee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:08.529919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d82c93d-da67-47c4-9749-53bf2b0790f8 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:08.550089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-70f2845b-1cac-4a0d-933e-dea82d7aacee None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:08.551526 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-70f2845b-1cac-4a0d-933e-dea82d7aacee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:08.608200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d89e86-0fb5-4043-bc94-52c78b403e29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:08.608860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d89e86-0fb5-4043-bc94-52c78b403e29 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:08.609142 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d89e86-0fb5-4043-bc94-52c78b403e29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:08.609416 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d89e86-0fb5-4043-bc94-52c78b403e29 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:08.610062 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d89e86-0fb5-4043-bc94-52c78b403e29 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:08.612800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d89e86-0fb5-4043-bc94-52c78b403e29 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:08.630924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1332/2696] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:08.640217 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b29ced-80ca-44c6-9793-8b7d4623243a None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:08.640723 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1365/2697] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:49:08 2022] POST /identity/v3/users => generated 293 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:49:08.640855 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b29ced-80ca-44c6-9793-8b7d4623243a None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:08.640942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b29ced-80ca-44c6-9793-8b7d4623243a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:08.644515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d89e86-0fb5-4043-bc94-52c78b403e29 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:08.646128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-22d89e86-0fb5-4043-bc94-52c78b403e29 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:08.659875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d89e86-0fb5-4043-bc94-52c78b403e29 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:08.667571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7afa155-5be0-45e0-a835-468f11652075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:08.668189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7afa155-5be0-45e0-a835-468f11652075 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:08.668547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7afa155-5be0-45e0-a835-468f11652075 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:08.668819 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7afa155-5be0-45e0-a835-468f11652075 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:08.669526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7afa155-5be0-45e0-a835-468f11652075 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:08.671327 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7afa155-5be0-45e0-a835-468f11652075 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:08.671988 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-43b29ced-80ca-44c6-9793-8b7d4623243a None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:08.673329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-43b29ced-80ca-44c6-9793-8b7d4623243a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:08.681657 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1333/2698] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:08.682914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1366/2699] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:08 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:49:08.686005 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b29ced-80ca-44c6-9793-8b7d4623243a None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:08.686478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7afa155-5be0-45e0-a835-468f11652075 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:08.693844 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:DELETE=1,keystone:SELECT=95,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:08.697725 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7afa155-5be0-45e0-a835-468f11652075 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:08.698162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b29ced-80ca-44c6-9793-8b7d4623243a None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:08.703281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae29ba0-b742-4c3c-af05-7de798995197 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:08.703888 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae29ba0-b742-4c3c-af05-7de798995197 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:08.704367 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae29ba0-b742-4c3c-af05-7de798995197 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:08.704597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae29ba0-b742-4c3c-af05-7de798995197 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:08.705256 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae29ba0-b742-4c3c-af05-7de798995197 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:49:08.705610 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae29ba0-b742-4c3c-af05-7de798995197 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:08.707722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae29ba0-b742-4c3c-af05-7de798995197 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:08.708699 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4815974-dc5e-42d9-918e-e9e34ef118a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:08.709368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4815974-dc5e-42d9-918e-e9e34ef118a0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:08.709710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4815974-dc5e-42d9-918e-e9e34ef118a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:08.710101 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4815974-dc5e-42d9-918e-e9e34ef118a0 admin admin] PATH_INFO: `/v3/projects/27f9bf030d2741eb852052dc2c16a381/users/98f92541a37e478db5d35ac2560301af/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:08.727735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1334/2700] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:49:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:49:08.729054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4815974-dc5e-42d9-918e-e9e34ef118a0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27f9bf030d2741eb852052dc2c16a381, user_id=98f92541a37e478db5d35ac2560301af, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:08.731739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4815974-dc5e-42d9-918e-e9e34ef118a0 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:08.732078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae29ba0-b742-4c3c-af05-7de798995197 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:08.744005 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae29ba0-b742-4c3c-af05-7de798995197 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:08.752924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1367/2701] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:08 2022] PUT /identity/v3/projects/27f9bf030d2741eb852052dc2c16a381/users/98f92541a37e478db5d35ac2560301af/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:49:08.757308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4815974-dc5e-42d9-918e-e9e34ef118a0 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:08.760934 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:08.825594 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbc5d04-8206-4562-a166-cb87e076a857 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:08.826191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbc5d04-8206-4562-a166-cb87e076a857 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:08.826467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbc5d04-8206-4562-a166-cb87e076a857 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:08.826735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbc5d04-8206-4562-a166-cb87e076a857 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:08.827388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbc5d04-8206-4562-a166-cb87e076a857 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:08.830765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbc5d04-8206-4562-a166-cb87e076a857 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:08.843900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4815974-dc5e-42d9-918e-e9e34ef118a0 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:08.848811 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa70be3d-2e43-4830-a284-455bd316c20f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:08.849507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa70be3d-2e43-4830-a284-455bd316c20f admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:08.849895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa70be3d-2e43-4830-a284-455bd316c20f admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:08.850195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa70be3d-2e43-4830-a284-455bd316c20f admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:08.851013 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa70be3d-2e43-4830-a284-455bd316c20f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:08.852767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa70be3d-2e43-4830-a284-455bd316c20f admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:08.862955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1335/2702] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:49:08 2022] POST /identity/v3/projects => generated 375 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:49:08.864666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1368/2703] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:08 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:49:08.864865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbc5d04-8206-4562-a166-cb87e076a857 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:08.869513 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa70be3d-2e43-4830-a284-455bd316c20f admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:08.879078 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa70be3d-2e43-4830-a284-455bd316c20f admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:08.950281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbc5d04-8206-4562-a166-cb87e076a857 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:08.953872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d629e292-53a7-41ee-8eb9-10fcf8b3398b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:08.954615 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d629e292-53a7-41ee-8eb9-10fcf8b3398b admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:08.954976 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d629e292-53a7-41ee-8eb9-10fcf8b3398b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:08.955338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d629e292-53a7-41ee-8eb9-10fcf8b3398b admin admin] PATH_INFO: `/v3/projects/27f9bf030d2741eb852052dc2c16a381/users/98f92541a37e478db5d35ac2560301af/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:08.955866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7778dca-2e70-444f-aaf4-6db87023e484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:08.957480 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7778dca-2e70-444f-aaf4-6db87023e484 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:08.958087 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7778dca-2e70-444f-aaf4-6db87023e484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:08.958440 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7778dca-2e70-444f-aaf4-6db87023e484 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:08.958821 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d629e292-53a7-41ee-8eb9-10fcf8b3398b admin admin] RBAC: Authorizing `identity:create_grant(project_id=27f9bf030d2741eb852052dc2c16a381, user_id=98f92541a37e478db5d35ac2560301af, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:08.959224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7778dca-2e70-444f-aaf4-6db87023e484 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:08.961351 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7778dca-2e70-444f-aaf4-6db87023e484 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:08.961809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d629e292-53a7-41ee-8eb9-10fcf8b3398b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:08.978194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1369/2704] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:08 2022] PUT /identity/v3/projects/27f9bf030d2741eb852052dc2c16a381/users/98f92541a37e478db5d35ac2560301af/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:49:08.989370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1336/2705] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:49:08 2022] POST /identity/v3/users => generated 288 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:49:08.993280 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d629e292-53a7-41ee-8eb9-10fcf8b3398b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:09.064728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d629e292-53a7-41ee-8eb9-10fcf8b3398b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:09.069582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f082b96-1a74-4a5e-a649-1ee583bce73e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:09.070247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f082b96-1a74-4a5e-a649-1ee583bce73e admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:09.070489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f082b96-1a74-4a5e-a649-1ee583bce73e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:09.070759 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f082b96-1a74-4a5e-a649-1ee583bce73e admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:09.071410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f082b96-1a74-4a5e-a649-1ee583bce73e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:09.073337 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f082b96-1a74-4a5e-a649-1ee583bce73e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:09.085850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1370/2706] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:08 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:09.092598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7778dca-2e70-444f-aaf4-6db87023e484 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:09.106406 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7778dca-2e70-444f-aaf4-6db87023e484 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:09.112860 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af5afcd-4c74-45b4-a8d0-27d8266b6d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:09.113495 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af5afcd-4c74-45b4-a8d0-27d8266b6d25 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:09.114140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af5afcd-4c74-45b4-a8d0-27d8266b6d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:09.114140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af5afcd-4c74-45b4-a8d0-27d8266b6d25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:09.114688 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af5afcd-4c74-45b4-a8d0-27d8266b6d25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:09.116366 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af5afcd-4c74-45b4-a8d0-27d8266b6d25 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:09.128346 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1337/2707] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:09 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:09.133068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f082b96-1a74-4a5e-a649-1ee583bce73e admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:09.147440 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f082b96-1a74-4a5e-a649-1ee583bce73e admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:09.155250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d85341-e194-480d-a4be-f65f52b02c6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:09.156073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d85341-e194-480d-a4be-f65f52b02c6a admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:09.156276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d85341-e194-480d-a4be-f65f52b02c6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:09.156702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d85341-e194-480d-a4be-f65f52b02c6a admin admin] PATH_INFO: `/v3/projects/81a954e41aee492390d703b330653728/users/34291935ced84832a77cbe0671ba4302/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:09.159660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:09.176485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d85341-e194-480d-a4be-f65f52b02c6a admin admin] RBAC: Authorizing `identity:create_grant(project_id=81a954e41aee492390d703b330653728, user_id=34291935ced84832a77cbe0671ba4302, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:09.179775 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d85341-e194-480d-a4be-f65f52b02c6a admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:09.198956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1371/2708] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:09 2022] PUT /identity/v3/projects/81a954e41aee492390d703b330653728/users/34291935ced84832a77cbe0671ba4302/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:09.209853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f03cb48-b180-432c-9805-c153b9cd53af None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:09.209853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f03cb48-b180-432c-9805-c153b9cd53af None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:09.211338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f03cb48-b180-432c-9805-c153b9cd53af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:09.247892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-2f03cb48-b180-432c-9805-c153b9cd53af None None] MFA Rules not processed for user `34291935ced84832a77cbe0671ba4302`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:09.249222 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2f03cb48-b180-432c-9805-c153b9cd53af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:09.320867 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1338/2709] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:09 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:09.335474 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d85341-e194-480d-a4be-f65f52b02c6a admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:09.337641 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b4d85341-e194-480d-a4be-f65f52b02c6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:09.418593 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d85341-e194-480d-a4be-f65f52b02c6a admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:09.423969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f41e1dc-d124-49e6-bdb2-cc20ac504ede service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '815504693bcc40b6aef06aacb2a9a35b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:09.424733 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f41e1dc-d124-49e6-bdb2-cc20ac504ede service glance] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:09.425086 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f41e1dc-d124-49e6-bdb2-cc20ac504ede service glance] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:09.425428 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f41e1dc-d124-49e6-bdb2-cc20ac504ede service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:09.436452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f41e1dc-d124-49e6-bdb2-cc20ac504ede service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:09.437935 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f41e1dc-d124-49e6-bdb2-cc20ac504ede service glance] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:09.450445 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1372/2710] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:09 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:49:11.448674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f03cb48-b180-432c-9805-c153b9cd53af None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:11.450102 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2f03cb48-b180-432c-9805-c153b9cd53af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:11.460821 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:11.551505 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f03cb48-b180-432c-9805-c153b9cd53af None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:11.557708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df97e3d-e1a9-4ff4-8f3d-d198c14a5e7a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:11.558116 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df97e3d-e1a9-4ff4-8f3d-d198c14a5e7a service placement] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:11.558470 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df97e3d-e1a9-4ff4-8f3d-d198c14a5e7a service placement] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:11.558974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df97e3d-e1a9-4ff4-8f3d-d198c14a5e7a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:11.561243 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9df97e3d-e1a9-4ff4-8f3d-d198c14a5e7a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:11.574965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df97e3d-e1a9-4ff4-8f3d-d198c14a5e7a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:11.576788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df97e3d-e1a9-4ff4-8f3d-d198c14a5e7a service placement] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:11.590827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1339/2711] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:49:13.980559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27db30d4-246b-47d6-a571-60f0cf584cea None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:13.981647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27db30d4-246b-47d6-a571-60f0cf584cea None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:13.981894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27db30d4-246b-47d6-a571-60f0cf584cea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:14.016839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-27db30d4-246b-47d6-a571-60f0cf584cea None None] MFA Rules not processed for user `98f92541a37e478db5d35ac2560301af`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:14.018194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-27db30d4-246b-47d6-a571-60f0cf584cea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:14.099556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1373/2712] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:13 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:14.117162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df97e3d-e1a9-4ff4-8f3d-d198c14a5e7a service placement] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:14.119014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9df97e3d-e1a9-4ff4-8f3d-d198c14a5e7a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:14.204750 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df97e3d-e1a9-4ff4-8f3d-d198c14a5e7a service placement] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:14.211608 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e55a7b6-31b7-477f-b4df-15adc9c30623 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:14.212628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e55a7b6-31b7-477f-b4df-15adc9c30623 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:14.213036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e55a7b6-31b7-477f-b4df-15adc9c30623 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:14.213321 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e55a7b6-31b7-477f-b4df-15adc9c30623 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:14.217489 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6e55a7b6-31b7-477f-b4df-15adc9c30623 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:14.301652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e55a7b6-31b7-477f-b4df-15adc9c30623 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:14.303319 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e55a7b6-31b7-477f-b4df-15adc9c30623 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:14.316967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1340/2713] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:14 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:49:15.405462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fac89e-05b1-4f7b-ad27-6051f1f62b42 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:15.405849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fac89e-05b1-4f7b-ad27-6051f1f62b42 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:15.405951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fac89e-05b1-4f7b-ad27-6051f1f62b42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:15.436334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-c6fac89e-05b1-4f7b-ad27-6051f1f62b42 None None] MFA Rules not processed for user `b287a26201c142f983d844c07df169e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:15.437671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c6fac89e-05b1-4f7b-ad27-6051f1f62b42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:15.447340 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1374/2714] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:15 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:24.191045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e55a7b6-31b7-477f-b4df-15adc9c30623 service nova] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:24.192504 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6e55a7b6-31b7-477f-b4df-15adc9c30623 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:24.230998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:24.249101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e55a7b6-31b7-477f-b4df-15adc9c30623 service nova] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:24.252801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c33129-1d36-41c4-8f22-ca9482a3c925 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:24.253449 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c33129-1d36-41c4-8f22-ca9482a3c925 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:24.253733 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c33129-1d36-41c4-8f22-ca9482a3c925 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:24.254001 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c33129-1d36-41c4-8f22-ca9482a3c925 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:24.255870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-28c33129-1d36-41c4-8f22-ca9482a3c925 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:24.319512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c33129-1d36-41c4-8f22-ca9482a3c925 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:24.321029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c33129-1d36-41c4-8f22-ca9482a3c925 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:24.335956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1341/2715] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:49:25.426819 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=70,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:27.538018 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fac89e-05b1-4f7b-ad27-6051f1f62b42 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:27.539486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c6fac89e-05b1-4f7b-ad27-6051f1f62b42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:27.553336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fac89e-05b1-4f7b-ad27-6051f1f62b42 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:27.558188 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7275d8be-3467-4a8c-9147-3333afd15a0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:27.558778 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7275d8be-3467-4a8c-9147-3333afd15a0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:27.559061 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7275d8be-3467-4a8c-9147-3333afd15a0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:27.559338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7275d8be-3467-4a8c-9147-3333afd15a0b admin admin] PATH_INFO: `/v3/users/4a8818fd59954480bcff68b9ba52769a` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:27.560576 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7275d8be-3467-4a8c-9147-3333afd15a0b admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a8818fd59954480bcff68b9ba52769a)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:27.562640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7275d8be-3467-4a8c-9147-3333afd15a0b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:27.617697 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7275d8be-3467-4a8c-9147-3333afd15a0b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a8818fd59954480bcff68b9ba52769a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:27.618526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7275d8be-3467-4a8c-9147-3333afd15a0b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a8818fd59954480bcff68b9ba52769a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:27.625986 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7275d8be-3467-4a8c-9147-3333afd15a0b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a8818fd59954480bcff68b9ba52769a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:27.626665 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7275d8be-3467-4a8c-9147-3333afd15a0b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a8818fd59954480bcff68b9ba52769a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:27.642760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7275d8be-3467-4a8c-9147-3333afd15a0b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a8818fd59954480bcff68b9ba52769a'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:27.656130 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1375/2716] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:49:27 2022] DELETE /identity/v3/users/4a8818fd59954480bcff68b9ba52769a => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:27.826154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c33129-1d36-41c4-8f22-ca9482a3c925 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:27.839218 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c33129-1d36-41c4-8f22-ca9482a3c925 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:27.904006 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:27.904577 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:27.904852 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:27.905120 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6 admin admin] PATH_INFO: `/v3/projects/81f514e581a3473188c0ca78416919a4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:27.906212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=81f514e581a3473188c0ca78416919a4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:27.908092 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:27.952401 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81f514e581a3473188c0ca78416919a4', 'request_id': 'req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:27.958433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81f514e581a3473188c0ca78416919a4', 'request_id': 'req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:27.965766 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81f514e581a3473188c0ca78416919a4', 'request_id': 'req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:27.976569 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6 admin admin] The token cache is being invalidate because project 81f514e581a3473188c0ca78416919a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:49:27.976854 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:27.977519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0f8f9a77-bab0-47e6-ae37-eeb7c9d7bae6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:27.978956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1342/2717] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:49:27 2022] DELETE /identity/v3/projects/81f514e581a3473188c0ca78416919a4 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:27.984686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185aacf0-03e2-4310-a5ff-2f645614feb9 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:27.984976 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185aacf0-03e2-4310-a5ff-2f645614feb9 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:27.985250 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185aacf0-03e2-4310-a5ff-2f645614feb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:28.005175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-185aacf0-03e2-4310-a5ff-2f645614feb9 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:28.006364 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-185aacf0-03e2-4310-a5ff-2f645614feb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:28.063486 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1376/2718] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:28.073351 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a72ed7-db9b-45c3-8152-a73b891a5735 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:28.073576 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a72ed7-db9b-45c3-8152-a73b891a5735 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:28.073859 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a72ed7-db9b-45c3-8152-a73b891a5735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:28.094099 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-e7a72ed7-db9b-45c3-8152-a73b891a5735 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:28.095289 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e7a72ed7-db9b-45c3-8152-a73b891a5735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:28.103672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1343/2719] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:28.107894 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185aacf0-03e2-4310-a5ff-2f645614feb9 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:28.117990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185aacf0-03e2-4310-a5ff-2f645614feb9 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:28.123326 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9a38fa-c139-47d4-9034-b827171dafce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:28.123936 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9a38fa-c139-47d4-9034-b827171dafce admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:28.124213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9a38fa-c139-47d4-9034-b827171dafce admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:28.124515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9a38fa-c139-47d4-9034-b827171dafce admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:28.125208 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9a38fa-c139-47d4-9034-b827171dafce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:49:28.125538 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9a38fa-c139-47d4-9034-b827171dafce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:28.127128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9a38fa-c139-47d4-9034-b827171dafce admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:28.140228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1377/2720] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:49:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:49:28.145151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a72ed7-db9b-45c3-8152-a73b891a5735 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:28.154211 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a72ed7-db9b-45c3-8152-a73b891a5735 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:28.158913 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4198ae-ca04-4b1c-a909-e9937ed54f13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:28.159556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4198ae-ca04-4b1c-a909-e9937ed54f13 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:28.159837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4198ae-ca04-4b1c-a909-e9937ed54f13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:28.160101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4198ae-ca04-4b1c-a909-e9937ed54f13 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:28.160788 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4198ae-ca04-4b1c-a909-e9937ed54f13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:28.162649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4198ae-ca04-4b1c-a909-e9937ed54f13 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:28.168188 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:28.195008 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1344/2721] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:49:28 2022] POST /identity/v3/projects => generated 365 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:49:28.199279 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9a38fa-c139-47d4-9034-b827171dafce admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:28.219589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9a38fa-c139-47d4-9034-b827171dafce admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:28.274681 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168dfc13-42c7-44c2-972c-0a4d2ae3c921 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:28.275274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168dfc13-42c7-44c2-972c-0a4d2ae3c921 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:28.275551 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168dfc13-42c7-44c2-972c-0a4d2ae3c921 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:28.275934 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168dfc13-42c7-44c2-972c-0a4d2ae3c921 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:28.276590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168dfc13-42c7-44c2-972c-0a4d2ae3c921 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:28.278492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168dfc13-42c7-44c2-972c-0a4d2ae3c921 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:28.286887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:28.306557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1378/2722] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:49:28 2022] POST /identity/v3/users => generated 283 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:49:28.308094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4198ae-ca04-4b1c-a909-e9937ed54f13 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:28.319091 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4198ae-ca04-4b1c-a909-e9937ed54f13 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:28.323857 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acad9ad-1a18-47aa-8f98-f345ed92c3c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:28.324428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acad9ad-1a18-47aa-8f98-f345ed92c3c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:28.324716 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acad9ad-1a18-47aa-8f98-f345ed92c3c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:28.324991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acad9ad-1a18-47aa-8f98-f345ed92c3c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:28.325634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acad9ad-1a18-47aa-8f98-f345ed92c3c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:28.327291 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acad9ad-1a18-47aa-8f98-f345ed92c3c3 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:28.338911 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1345/2723] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:28.342949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168dfc13-42c7-44c2-972c-0a4d2ae3c921 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:28.353130 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168dfc13-42c7-44c2-972c-0a4d2ae3c921 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:28.358703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0bf813-9493-4f7f-afcd-c415985e8d8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:28.359290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0bf813-9493-4f7f-afcd-c415985e8d8c admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:28.359568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0bf813-9493-4f7f-afcd-c415985e8d8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:28.359837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0bf813-9493-4f7f-afcd-c415985e8d8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:28.360527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0bf813-9493-4f7f-afcd-c415985e8d8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:28.362393 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0bf813-9493-4f7f-afcd-c415985e8d8c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:28.374519 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1379/2724] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:28 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:28.379602 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acad9ad-1a18-47aa-8f98-f345ed92c3c3 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:28.390003 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acad9ad-1a18-47aa-8f98-f345ed92c3c3 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:28.394722 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c52a1f-184d-46bd-afc5-f66625730d52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:28.395380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c52a1f-184d-46bd-afc5-f66625730d52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:28.395694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c52a1f-184d-46bd-afc5-f66625730d52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:28.395972 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c52a1f-184d-46bd-afc5-f66625730d52 admin admin] PATH_INFO: `/v3/projects/4715d1a2e1ab4fc2aca83e84941729bd/users/0d1427409a004a4c9416fea87f7a661f/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:28.413738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c52a1f-184d-46bd-afc5-f66625730d52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4715d1a2e1ab4fc2aca83e84941729bd, user_id=0d1427409a004a4c9416fea87f7a661f, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:28.416710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c52a1f-184d-46bd-afc5-f66625730d52 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:28.429547 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1346/2725] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:28 2022] PUT /identity/v3/projects/4715d1a2e1ab4fc2aca83e84941729bd/users/0d1427409a004a4c9416fea87f7a661f/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:28.440051 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b830087-0c3d-415b-8bae-2ae3d93b4378 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:28.440379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b830087-0c3d-415b-8bae-2ae3d93b4378 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:28.440663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b830087-0c3d-415b-8bae-2ae3d93b4378 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:28.468415 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-1b830087-0c3d-415b-8bae-2ae3d93b4378 None None] MFA Rules not processed for user `0d1427409a004a4c9416fea87f7a661f`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:28.469660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1b830087-0c3d-415b-8bae-2ae3d93b4378 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:28.535548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1380/2726] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:28 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:28.546113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c52a1f-184d-46bd-afc5-f66625730d52 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:28.547804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d4c52a1f-184d-46bd-afc5-f66625730d52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:28.616596 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c52a1f-184d-46bd-afc5-f66625730d52 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:28.620171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d53381b-a8b0-4327-b626-40fb2c144842 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:28.620803 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d53381b-a8b0-4327-b626-40fb2c144842 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:28.621082 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d53381b-a8b0-4327-b626-40fb2c144842 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:28.621351 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d53381b-a8b0-4327-b626-40fb2c144842 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:28.630235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d53381b-a8b0-4327-b626-40fb2c144842 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:28.631675 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d53381b-a8b0-4327-b626-40fb2c144842 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:28.644579 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1347/2727] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:28 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:49:38.532801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:INSERT=3,keystone:SELECT=39 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:38.640470 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=51,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:50.782304 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b830087-0c3d-415b-8bae-2ae3d93b4378 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:50.842752 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b830087-0c3d-415b-8bae-2ae3d93b4378 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:50.847360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7a9d26-b0a2-47f9-a48a-f2a1e91f768c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:50.847996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7a9d26-b0a2-47f9-a48a-f2a1e91f768c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:50.848284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7a9d26-b0a2-47f9-a48a-f2a1e91f768c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:50.848596 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7a9d26-b0a2-47f9-a48a-f2a1e91f768c admin admin] PATH_INFO: `/v3/users/0d1427409a004a4c9416fea87f7a661f` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:50.849726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7a9d26-b0a2-47f9-a48a-f2a1e91f768c admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d1427409a004a4c9416fea87f7a661f)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:50.852222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7a9d26-b0a2-47f9-a48a-f2a1e91f768c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:50.897582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7c7a9d26-b0a2-47f9-a48a-f2a1e91f768c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d1427409a004a4c9416fea87f7a661f'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:50.898740 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7c7a9d26-b0a2-47f9-a48a-f2a1e91f768c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d1427409a004a4c9416fea87f7a661f'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:50.905927 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7c7a9d26-b0a2-47f9-a48a-f2a1e91f768c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d1427409a004a4c9416fea87f7a661f'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:50.906548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7c7a9d26-b0a2-47f9-a48a-f2a1e91f768c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d1427409a004a4c9416fea87f7a661f'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:50.918919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-7c7a9d26-b0a2-47f9-a48a-f2a1e91f768c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d1427409a004a4c9416fea87f7a661f'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:50.931896 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1381/2728] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:49:50 2022] DELETE /identity/v3/users/0d1427409a004a4c9416fea87f7a661f => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:50.938313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e920456c-9ef1-4bfd-9fe8-00109f0ad866 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:50.938781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e920456c-9ef1-4bfd-9fe8-00109f0ad866 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:50.938901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e920456c-9ef1-4bfd-9fe8-00109f0ad866 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:50.957883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-e920456c-9ef1-4bfd-9fe8-00109f0ad866 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:50.959079 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e920456c-9ef1-4bfd-9fe8-00109f0ad866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:51.028698 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1348/2729] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:51.038149 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7a9d26-b0a2-47f9-a48a-f2a1e91f768c admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:51.039786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7c7a9d26-b0a2-47f9-a48a-f2a1e91f768c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:51.096702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7a9d26-b0a2-47f9-a48a-f2a1e91f768c admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:51.100315 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f61a292-a7e7-45c6-9a82-8ae63485da0f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:51.100938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f61a292-a7e7-45c6-9a82-8ae63485da0f service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:51.101214 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f61a292-a7e7-45c6-9a82-8ae63485da0f service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:51.101485 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f61a292-a7e7-45c6-9a82-8ae63485da0f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:51.110517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f61a292-a7e7-45c6-9a82-8ae63485da0f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:51.112194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f61a292-a7e7-45c6-9a82-8ae63485da0f service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:51.121514 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1382/2730] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:49:51.389554 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e920456c-9ef1-4bfd-9fe8-00109f0ad866 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:51.391438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e920456c-9ef1-4bfd-9fe8-00109f0ad866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:51.404166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e920456c-9ef1-4bfd-9fe8-00109f0ad866 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:51.408917 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:51.409551 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:51.409831 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:51.410105 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5 admin admin] PATH_INFO: `/v3/projects/4715d1a2e1ab4fc2aca83e84941729bd` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:51.411418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4715d1a2e1ab4fc2aca83e84941729bd)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:51.413507 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:51.469841 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4715d1a2e1ab4fc2aca83e84941729bd', 'request_id': 'req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:51.475425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4715d1a2e1ab4fc2aca83e84941729bd', 'request_id': 'req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:51.484910 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4715d1a2e1ab4fc2aca83e84941729bd', 'request_id': 'req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:51.499930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5 admin admin] The token cache is being invalidate because project 4715d1a2e1ab4fc2aca83e84941729bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:49:51.500365 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:51.501301 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-05ae177b-220d-4ce6-bd2f-2ba90da2c3e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:51.502774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1349/2731] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:49:51 2022] DELETE /identity/v3/projects/4715d1a2e1ab4fc2aca83e84941729bd => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:51.508850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e823efa-fd88-4ae1-91fa-07a834a65ba7 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:51.509185 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e823efa-fd88-4ae1-91fa-07a834a65ba7 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:51.509442 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e823efa-fd88-4ae1-91fa-07a834a65ba7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:51.529958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-3e823efa-fd88-4ae1-91fa-07a834a65ba7 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:51.531164 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-3e823efa-fd88-4ae1-91fa-07a834a65ba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:51.548106 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:51.591562 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1383/2732] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:51.600853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f92aea-dc6d-440c-a8d6-2b3b10005a77 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:51.601136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f92aea-dc6d-440c-a8d6-2b3b10005a77 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:51.602317 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f92aea-dc6d-440c-a8d6-2b3b10005a77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:51.623770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-88f92aea-dc6d-440c-a8d6-2b3b10005a77 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:51.624992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-88f92aea-dc6d-440c-a8d6-2b3b10005a77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:51.633314 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1350/2733] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:51.691379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e823efa-fd88-4ae1-91fa-07a834a65ba7 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:51.701700 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e823efa-fd88-4ae1-91fa-07a834a65ba7 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:51.706178 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10b8caa-3b75-4111-9987-853ca10f44e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:51.706768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10b8caa-3b75-4111-9987-853ca10f44e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:51.707040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10b8caa-3b75-4111-9987-853ca10f44e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:51.707309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10b8caa-3b75-4111-9987-853ca10f44e6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:51.707952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10b8caa-3b75-4111-9987-853ca10f44e6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:49:51.708251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10b8caa-3b75-4111-9987-853ca10f44e6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:51.709847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10b8caa-3b75-4111-9987-853ca10f44e6 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:51.721072 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1384/2734] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:49:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:49:51.725290 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f92aea-dc6d-440c-a8d6-2b3b10005a77 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:51.734823 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f92aea-dc6d-440c-a8d6-2b3b10005a77 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:51.739711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9425a9dd-670a-490f-91a6-1c3953ef7442 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:51.740297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9425a9dd-670a-490f-91a6-1c3953ef7442 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:51.740599 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9425a9dd-670a-490f-91a6-1c3953ef7442 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:51.740871 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9425a9dd-670a-490f-91a6-1c3953ef7442 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:51.741501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9425a9dd-670a-490f-91a6-1c3953ef7442 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:51.744130 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9425a9dd-670a-490f-91a6-1c3953ef7442 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:51.893866 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1351/2735] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:49:51 2022] POST /identity/v3/projects => generated 365 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:49:51.898238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10b8caa-3b75-4111-9987-853ca10f44e6 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:51.908495 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10b8caa-3b75-4111-9987-853ca10f44e6 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:51.964236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14f7ce8-fc41-4807-bed8-93b8f7ad1e86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:51.964854 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14f7ce8-fc41-4807-bed8-93b8f7ad1e86 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:51.965133 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14f7ce8-fc41-4807-bed8-93b8f7ad1e86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:51.965403 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14f7ce8-fc41-4807-bed8-93b8f7ad1e86 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:51.966030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14f7ce8-fc41-4807-bed8-93b8f7ad1e86 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:51.967940 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14f7ce8-fc41-4807-bed8-93b8f7ad1e86 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:51.989225 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1385/2736] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:49:51 2022] POST /identity/v3/users => generated 283 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:49:51.994017 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9425a9dd-670a-490f-91a6-1c3953ef7442 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.004195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9425a9dd-670a-490f-91a6-1c3953ef7442 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.009051 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5cb3ee-52b5-44bf-9e0a-8f8bc0f764e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.009647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5cb3ee-52b5-44bf-9e0a-8f8bc0f764e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.009929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5cb3ee-52b5-44bf-9e0a-8f8bc0f764e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.010203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5cb3ee-52b5-44bf-9e0a-8f8bc0f764e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.010843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5cb3ee-52b5-44bf-9e0a-8f8bc0f764e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.012519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5cb3ee-52b5-44bf-9e0a-8f8bc0f764e6 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:52.024369 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1352/2737] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:52.028338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14f7ce8-fc41-4807-bed8-93b8f7ad1e86 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.038383 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14f7ce8-fc41-4807-bed8-93b8f7ad1e86 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.043412 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef260dd-25be-42be-ad63-49b3bd47ccd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.043993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef260dd-25be-42be-ad63-49b3bd47ccd5 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.044274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef260dd-25be-42be-ad63-49b3bd47ccd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.044651 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef260dd-25be-42be-ad63-49b3bd47ccd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.045288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef260dd-25be-42be-ad63-49b3bd47ccd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.046853 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef260dd-25be-42be-ad63-49b3bd47ccd5 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:52.061165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1386/2738] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:52 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:52.062290 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5cb3ee-52b5-44bf-9e0a-8f8bc0f764e6 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.068870 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef260dd-25be-42be-ad63-49b3bd47ccd5 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.070568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6ef260dd-25be-42be-ad63-49b3bd47ccd5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:52.073063 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5cb3ee-52b5-44bf-9e0a-8f8bc0f764e6 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.077608 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99597eed-4a15-44d3-ad26-ead6e33116fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.078248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99597eed-4a15-44d3-ad26-ead6e33116fb admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.078524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99597eed-4a15-44d3-ad26-ead6e33116fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.078804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99597eed-4a15-44d3-ad26-ead6e33116fb admin admin] PATH_INFO: `/v3/projects/9e57e45509e64231be32e5d652c57730/users/d2e9c6abc36741c5a02db3dcf7505584/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.082938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef260dd-25be-42be-ad63-49b3bd47ccd5 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.087969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b874e683-6d5e-4f98-880e-a86ab681430b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.088604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b874e683-6d5e-4f98-880e-a86ab681430b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.088883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b874e683-6d5e-4f98-880e-a86ab681430b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.089155 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b874e683-6d5e-4f98-880e-a86ab681430b admin admin] PATH_INFO: `/v3/users/693b2af1eda24bc3bb641b5724b7acac` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.090279 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b874e683-6d5e-4f98-880e-a86ab681430b admin admin] RBAC: Authorizing `identity:delete_user(user_id=693b2af1eda24bc3bb641b5724b7acac)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.092158 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b874e683-6d5e-4f98-880e-a86ab681430b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:52.094112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99597eed-4a15-44d3-ad26-ead6e33116fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e57e45509e64231be32e5d652c57730, user_id=d2e9c6abc36741c5a02db3dcf7505584, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.096753 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99597eed-4a15-44d3-ad26-ead6e33116fb admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:52.111061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1353/2739] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:52 2022] PUT /identity/v3/projects/9e57e45509e64231be32e5d652c57730/users/d2e9c6abc36741c5a02db3dcf7505584/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:52.121972 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a24963-6ccd-47e9-908c-47f0d7169897 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.122255 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a24963-6ccd-47e9-908c-47f0d7169897 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.122528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a24963-6ccd-47e9-908c-47f0d7169897 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.140247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-b874e683-6d5e-4f98-880e-a86ab681430b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '693b2af1eda24bc3bb641b5724b7acac'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:52.141010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-b874e683-6d5e-4f98-880e-a86ab681430b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '693b2af1eda24bc3bb641b5724b7acac'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:52.148731 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-b874e683-6d5e-4f98-880e-a86ab681430b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '693b2af1eda24bc3bb641b5724b7acac'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:52.149391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-b874e683-6d5e-4f98-880e-a86ab681430b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '693b2af1eda24bc3bb641b5724b7acac'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:52.150762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-c1a24963-6ccd-47e9-908c-47f0d7169897 None None] MFA Rules not processed for user `d2e9c6abc36741c5a02db3dcf7505584`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:52.151990 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c1a24963-6ccd-47e9-908c-47f0d7169897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:52.160294 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:52.161658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-b874e683-6d5e-4f98-880e-a86ab681430b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '693b2af1eda24bc3bb641b5724b7acac'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:52.171612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:52.176885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1387/2740] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:49:52 2022] DELETE /identity/v3/users/693b2af1eda24bc3bb641b5724b7acac => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:52.182128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0283e8-dcf2-4586-af5c-41d490db750f None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.182669 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0283e8-dcf2-4586-af5c-41d490db750f None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.183203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0283e8-dcf2-4586-af5c-41d490db750f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.208655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-fa0283e8-dcf2-4586-af5c-41d490db750f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:52.209884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-fa0283e8-dcf2-4586-af5c-41d490db750f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:52.239147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1354/2741] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:52 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:49:52.244020 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a24963-6ccd-47e9-908c-47f0d7169897 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.245433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c1a24963-6ccd-47e9-908c-47f0d7169897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:52.271494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1388/2742] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:49:52.281715 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0283e8-dcf2-4586-af5c-41d490db750f None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.283134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-fa0283e8-dcf2-4586-af5c-41d490db750f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:52.310726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a24963-6ccd-47e9-908c-47f0d7169897 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.315380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12309ddf-33dc-49d1-b810-ce65197f47a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.315989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12309ddf-33dc-49d1-b810-ce65197f47a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.316267 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12309ddf-33dc-49d1-b810-ce65197f47a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.316573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12309ddf-33dc-49d1-b810-ce65197f47a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.317195 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12309ddf-33dc-49d1-b810-ce65197f47a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.319089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12309ddf-33dc-49d1-b810-ce65197f47a0 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:52.340772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0283e8-dcf2-4586-af5c-41d490db750f None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.344441 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c1e091-e784-4407-b838-cde269e599e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.345013 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c1e091-e784-4407-b838-cde269e599e6 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.345290 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c1e091-e784-4407-b838-cde269e599e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.345564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c1e091-e784-4407-b838-cde269e599e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.351022 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1355/2743] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:49:52 2022] POST /identity/v3/projects => generated 365 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:49:52.354965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12309ddf-33dc-49d1-b810-ce65197f47a0 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.356762 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c1e091-e784-4407-b838-cde269e599e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.358367 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c1e091-e784-4407-b838-cde269e599e6 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:52.369849 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1389/2744] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:49:52.422762 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12309ddf-33dc-49d1-b810-ce65197f47a0 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.427616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa53a812-ddf3-457c-aa10-9a3a1a89f51e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.428247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa53a812-ddf3-457c-aa10-9a3a1a89f51e admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.428524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa53a812-ddf3-457c-aa10-9a3a1a89f51e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.428791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa53a812-ddf3-457c-aa10-9a3a1a89f51e admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.429955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa53a812-ddf3-457c-aa10-9a3a1a89f51e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.431886 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa53a812-ddf3-457c-aa10-9a3a1a89f51e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:52.453996 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1356/2745] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:49:52 2022] POST /identity/v3/users => generated 289 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:49:52.457996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c1e091-e784-4407-b838-cde269e599e6 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.468111 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c1e091-e784-4407-b838-cde269e599e6 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.473613 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805ed516-6bae-45c7-b8b9-2e4b919dbbf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.474438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805ed516-6bae-45c7-b8b9-2e4b919dbbf9 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.474720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805ed516-6bae-45c7-b8b9-2e4b919dbbf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.474988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805ed516-6bae-45c7-b8b9-2e4b919dbbf9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.475626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805ed516-6bae-45c7-b8b9-2e4b919dbbf9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.477741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805ed516-6bae-45c7-b8b9-2e4b919dbbf9 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:52.489189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1390/2746] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:49:52.498564 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa53a812-ddf3-457c-aa10-9a3a1a89f51e admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.508690 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa53a812-ddf3-457c-aa10-9a3a1a89f51e admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.513496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc1dd5a-cf3a-4883-89e9-e150a9856b1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.514161 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc1dd5a-cf3a-4883-89e9-e150a9856b1c admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.514443 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc1dd5a-cf3a-4883-89e9-e150a9856b1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.514721 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc1dd5a-cf3a-4883-89e9-e150a9856b1c admin admin] PATH_INFO: `/v3/projects/d84cb8858c304f87be0789a2afd3e211/users/0fa102bbaf6e446687157d8861e95561/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.535281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc1dd5a-cf3a-4883-89e9-e150a9856b1c admin admin] RBAC: Authorizing `identity:create_grant(project_id=d84cb8858c304f87be0789a2afd3e211, user_id=0fa102bbaf6e446687157d8861e95561, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.538010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc1dd5a-cf3a-4883-89e9-e150a9856b1c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:52.551452 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:52.559274 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1357/2747] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:52 2022] PUT /identity/v3/projects/d84cb8858c304f87be0789a2afd3e211/users/0fa102bbaf6e446687157d8861e95561/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:52.564674 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805ed516-6bae-45c7-b8b9-2e4b919dbbf9 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.575319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805ed516-6bae-45c7-b8b9-2e4b919dbbf9 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.635487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149bb54b-2e5b-43bd-9fa0-f729c67a9fab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.636093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149bb54b-2e5b-43bd-9fa0-f729c67a9fab admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.636425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149bb54b-2e5b-43bd-9fa0-f729c67a9fab admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.636702 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149bb54b-2e5b-43bd-9fa0-f729c67a9fab admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.637341 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149bb54b-2e5b-43bd-9fa0-f729c67a9fab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.639011 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149bb54b-2e5b-43bd-9fa0-f729c67a9fab admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:52.649357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1391/2748] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:52 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:52.654018 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc1dd5a-cf3a-4883-89e9-e150a9856b1c admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.659703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149bb54b-2e5b-43bd-9fa0-f729c67a9fab admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.663179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-149bb54b-2e5b-43bd-9fa0-f729c67a9fab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:52.668244 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc1dd5a-cf3a-4883-89e9-e150a9856b1c admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.673976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e5ebb2-b0a9-4430-8a14-36681d2de313 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.674638 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e5ebb2-b0a9-4430-8a14-36681d2de313 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.675060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e5ebb2-b0a9-4430-8a14-36681d2de313 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.675182 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e5ebb2-b0a9-4430-8a14-36681d2de313 admin admin] PATH_INFO: `/v3/projects/d84cb8858c304f87be0789a2afd3e211/users/0fa102bbaf6e446687157d8861e95561/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.677153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e5ebb2-b0a9-4430-8a14-36681d2de313 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d84cb8858c304f87be0789a2afd3e211, user_id=0fa102bbaf6e446687157d8861e95561, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.680952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e5ebb2-b0a9-4430-8a14-36681d2de313 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:52.682964 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149bb54b-2e5b-43bd-9fa0-f729c67a9fab admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.688171 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.688972 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.689130 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.689379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73 admin admin] PATH_INFO: `/v3/projects/03576185db7b431d9e5d5cd25bb5235b` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.690594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=03576185db7b431d9e5d5cd25bb5235b)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.692522 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:52.696465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1358/2749] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:52 2022] PUT /identity/v3/projects/d84cb8858c304f87be0789a2afd3e211/users/0fa102bbaf6e446687157d8861e95561/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:52.706664 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac76591-8bee-4e16-8c0d-918af2facce7 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.707167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac76591-8bee-4e16-8c0d-918af2facce7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.707445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac76591-8bee-4e16-8c0d-918af2facce7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.722317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:52.741634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-5ac76591-8bee-4e16-8c0d-918af2facce7 None None] MFA Rules not processed for user `0fa102bbaf6e446687157d8861e95561`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:52.743508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5ac76591-8bee-4e16-8c0d-918af2facce7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:52.750278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '03576185db7b431d9e5d5cd25bb5235b', 'request_id': 'req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:52.756339 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '03576185db7b431d9e5d5cd25bb5235b', 'request_id': 'req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:52.763898 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '03576185db7b431d9e5d5cd25bb5235b', 'request_id': 'req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:52.775395 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73 admin admin] The token cache is being invalidate because project 03576185db7b431d9e5d5cd25bb5235b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:49:52.775686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:52.776376 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-786c820c-5464-49e3-b2e1-1b5e5fc1fa73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:52.778005 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1392/2750] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:49:52 2022] DELETE /identity/v3/projects/03576185db7b431d9e5d5cd25bb5235b => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:49:52.784646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e81ef3e-5d13-4c56-8c47-4cc8f0210a0d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.784962 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e81ef3e-5d13-4c56-8c47-4cc8f0210a0d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.785580 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e81ef3e-5d13-4c56-8c47-4cc8f0210a0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.805582 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-5e81ef3e-5d13-4c56-8c47-4cc8f0210a0d None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:52.806783 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5e81ef3e-5d13-4c56-8c47-4cc8f0210a0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:52.813471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1359/2751] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:52 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:49:52.824929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac76591-8bee-4e16-8c0d-918af2facce7 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.826393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5ac76591-8bee-4e16-8c0d-918af2facce7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:52.872995 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1393/2752] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:49:52.881276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15a2ee1-f5df-44d6-b6fa-96b398e829ec None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.881573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15a2ee1-f5df-44d6-b6fa-96b398e829ec None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.881885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15a2ee1-f5df-44d6-b6fa-96b398e829ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.893763 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac76591-8bee-4e16-8c0d-918af2facce7 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.898639 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9439d93-b37b-42b6-884a-046a10c9a3f3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.899259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9439d93-b37b-42b6-884a-046a10c9a3f3 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.899539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9439d93-b37b-42b6-884a-046a10c9a3f3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.899809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9439d93-b37b-42b6-884a-046a10c9a3f3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.902247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-d15a2ee1-f5df-44d6-b6fa-96b398e829ec None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:52.903426 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d15a2ee1-f5df-44d6-b6fa-96b398e829ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:52.909611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9439d93-b37b-42b6-884a-046a10c9a3f3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.911212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9439d93-b37b-42b6-884a-046a10c9a3f3 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:52.917536 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1394/2753] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:52.917779 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15a2ee1-f5df-44d6-b6fa-96b398e829ec None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.924512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1360/2754] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:52 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:49:52.931067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15a2ee1-f5df-44d6-b6fa-96b398e829ec None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.936774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47afae19-7eb5-4667-87f5-0c5770a053c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.937531 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47afae19-7eb5-4667-87f5-0c5770a053c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.938270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47afae19-7eb5-4667-87f5-0c5770a053c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.938621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47afae19-7eb5-4667-87f5-0c5770a053c7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.939287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47afae19-7eb5-4667-87f5-0c5770a053c7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:49:52.939579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47afae19-7eb5-4667-87f5-0c5770a053c7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.941176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47afae19-7eb5-4667-87f5-0c5770a053c7 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:52.953443 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1395/2755] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:49:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:49:52.959045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9439d93-b37b-42b6-884a-046a10c9a3f3 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:52.974922 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9439d93-b37b-42b6-884a-046a10c9a3f3 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:52.980432 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8aa1e9-acb5-45e9-87c7-e3c25e6fba17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:52.981084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8aa1e9-acb5-45e9-87c7-e3c25e6fba17 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:52.981333 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8aa1e9-acb5-45e9-87c7-e3c25e6fba17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:52.981600 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8aa1e9-acb5-45e9-87c7-e3c25e6fba17 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:52.982229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8aa1e9-acb5-45e9-87c7-e3c25e6fba17 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:52.984109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8aa1e9-acb5-45e9-87c7-e3c25e6fba17 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:53.006254 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47afae19-7eb5-4667-87f5-0c5770a053c7 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:53.015194 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1361/2756] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:49:52 2022] POST /identity/v3/projects => generated 379 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:49:53.017407 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47afae19-7eb5-4667-87f5-0c5770a053c7 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:53.019058 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8aa1e9-acb5-45e9-87c7-e3c25e6fba17 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:53.069009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2a0098-4836-4ce6-9436-b85096f56601 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:53.069617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2a0098-4836-4ce6-9436-b85096f56601 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:53.069938 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2a0098-4836-4ce6-9436-b85096f56601 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:53.070764 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2a0098-4836-4ce6-9436-b85096f56601 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:53.072299 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:53.072612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6e2a0098-4836-4ce6-9436-b85096f56601 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:53.084235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8aa1e9-acb5-45e9-87c7-e3c25e6fba17 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:53.089287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bd543f-0e1d-4e13-9b6b-d851b8d0ecb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:53.089879 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bd543f-0e1d-4e13-9b6b-d851b8d0ecb2 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:53.090589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bd543f-0e1d-4e13-9b6b-d851b8d0ecb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:53.090864 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bd543f-0e1d-4e13-9b6b-d851b8d0ecb2 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:53.091516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5bd543f-0e1d-4e13-9b6b-d851b8d0ecb2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:53.093490 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5bd543f-0e1d-4e13-9b6b-d851b8d0ecb2 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:53.115108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1362/2757] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:49:53 2022] POST /identity/v3/users => generated 290 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:49:53.119538 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bd543f-0e1d-4e13-9b6b-d851b8d0ecb2 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:53.120275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:53.129637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bd543f-0e1d-4e13-9b6b-d851b8d0ecb2 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:53.135232 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33675c67-c512-4181-8f0d-6446043105be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:53.135812 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33675c67-c512-4181-8f0d-6446043105be admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:53.136087 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33675c67-c512-4181-8f0d-6446043105be admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:53.136388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33675c67-c512-4181-8f0d-6446043105be admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:53.137167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33675c67-c512-4181-8f0d-6446043105be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:53.138755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33675c67-c512-4181-8f0d-6446043105be admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:53.140201 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2a0098-4836-4ce6-9436-b85096f56601 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:53.141671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2a0098-4836-4ce6-9436-b85096f56601 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:53.148488 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1363/2758] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:53 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:53.151650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1396/2759] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:53 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:49:53.152417 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33675c67-c512-4181-8f0d-6446043105be admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:53.161941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33675c67-c512-4181-8f0d-6446043105be admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:53.166558 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e29473-fc5b-4626-9d90-08b88d41c586 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:53.167146 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e29473-fc5b-4626-9d90-08b88d41c586 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:53.167423 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e29473-fc5b-4626-9d90-08b88d41c586 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:53.167692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e29473-fc5b-4626-9d90-08b88d41c586 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:53.168357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e29473-fc5b-4626-9d90-08b88d41c586 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:53.169938 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e29473-fc5b-4626-9d90-08b88d41c586 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:53.183708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1364/2760] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:53 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:49:53.188093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2a0098-4836-4ce6-9436-b85096f56601 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:53.199135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2a0098-4836-4ce6-9436-b85096f56601 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:53.204041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8485bae1-394d-4422-b194-b76d2b841dfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:53.204723 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8485bae1-394d-4422-b194-b76d2b841dfe admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:53.205000 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8485bae1-394d-4422-b194-b76d2b841dfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:53.205270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8485bae1-394d-4422-b194-b76d2b841dfe admin admin] PATH_INFO: `/v3/projects/1c8a6694cad9480f86dc4dca113bb762/users/4d70d61439324d4b9c3b3513799b896c/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:53.223874 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8485bae1-394d-4422-b194-b76d2b841dfe admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c8a6694cad9480f86dc4dca113bb762, user_id=4d70d61439324d4b9c3b3513799b896c, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:53.229293 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8485bae1-394d-4422-b194-b76d2b841dfe admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:53.244263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1397/2761] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:53 2022] PUT /identity/v3/projects/1c8a6694cad9480f86dc4dca113bb762/users/4d70d61439324d4b9c3b3513799b896c/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:49:53.249414 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a7092d-3b2f-4ff5-9577-48a48159799c None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:53.249685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a7092d-3b2f-4ff5-9577-48a48159799c None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:53.249960 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a7092d-3b2f-4ff5-9577-48a48159799c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:53.275115 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-32a7092d-3b2f-4ff5-9577-48a48159799c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:53.276352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-32a7092d-3b2f-4ff5-9577-48a48159799c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:53.338205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1365/2762] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:49:53.347612 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8485bae1-394d-4422-b194-b76d2b841dfe admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:53.349463 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8485bae1-394d-4422-b194-b76d2b841dfe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:53.405798 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8485bae1-394d-4422-b194-b76d2b841dfe admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:53.409459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc6dea5-d5f9-4583-a7a8-f3621da08ccf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:53.410084 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc6dea5-d5f9-4583-a7a8-f3621da08ccf service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:53.410363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc6dea5-d5f9-4583-a7a8-f3621da08ccf service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:53.410635 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc6dea5-d5f9-4583-a7a8-f3621da08ccf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:53.419613 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc6dea5-d5f9-4583-a7a8-f3621da08ccf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:53.421103 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc6dea5-d5f9-4583-a7a8-f3621da08ccf service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:53.432425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1398/2763] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:49:56.017645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a7092d-3b2f-4ff5-9577-48a48159799c None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:56.019891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-32a7092d-3b2f-4ff5-9577-48a48159799c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:56.032198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a7092d-3b2f-4ff5-9577-48a48159799c None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:56.037089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e42e9f0-749d-4c63-9ace-fee73f9a12c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:56.037744 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e42e9f0-749d-4c63-9ace-fee73f9a12c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:56.038025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e42e9f0-749d-4c63-9ace-fee73f9a12c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:56.038330 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e42e9f0-749d-4c63-9ace-fee73f9a12c4 admin admin] PATH_INFO: `/v3/users/d2e9c6abc36741c5a02db3dcf7505584` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:56.041771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e42e9f0-749d-4c63-9ace-fee73f9a12c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2e9c6abc36741c5a02db3dcf7505584)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:56.043714 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e42e9f0-749d-4c63-9ace-fee73f9a12c4 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:56.102436 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-9e42e9f0-749d-4c63-9ace-fee73f9a12c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2e9c6abc36741c5a02db3dcf7505584'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:56.102992 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-9e42e9f0-749d-4c63-9ace-fee73f9a12c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2e9c6abc36741c5a02db3dcf7505584'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:56.104792 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-9e42e9f0-749d-4c63-9ace-fee73f9a12c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2e9c6abc36741c5a02db3dcf7505584'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:56.118438 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-9e42e9f0-749d-4c63-9ace-fee73f9a12c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2e9c6abc36741c5a02db3dcf7505584'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:56.147213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-9e42e9f0-749d-4c63-9ace-fee73f9a12c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2e9c6abc36741c5a02db3dcf7505584'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:56.164819 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1366/2764] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:49:56 2022] DELETE /identity/v3/users/d2e9c6abc36741c5a02db3dcf7505584 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:56.169409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7ee14f-a332-476d-bcb9-0d80a4d096d7 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:56.169955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7ee14f-a332-476d-bcb9-0d80a4d096d7 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:56.169955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7ee14f-a332-476d-bcb9-0d80a4d096d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:56.193841 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-0c7ee14f-a332-476d-bcb9-0d80a4d096d7 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:56.195134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0c7ee14f-a332-476d-bcb9-0d80a4d096d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:56.278259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1399/2765] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:49:56.286944 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e42e9f0-749d-4c63-9ace-fee73f9a12c4 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:56.289626 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9e42e9f0-749d-4c63-9ace-fee73f9a12c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:56.354897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:56.387286 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e42e9f0-749d-4c63-9ace-fee73f9a12c4 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:56.393980 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7748628-b360-4888-81de-f1c66c806f8a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:56.395703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7748628-b360-4888-81de-f1c66c806f8a service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:56.396192 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7748628-b360-4888-81de-f1c66c806f8a service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:56.397156 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7748628-b360-4888-81de-f1c66c806f8a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:56.409423 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7748628-b360-4888-81de-f1c66c806f8a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:56.410878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7748628-b360-4888-81de-f1c66c806f8a service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:56.431877 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1367/2766] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:49:56.923887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7ee14f-a332-476d-bcb9-0d80a4d096d7 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:56.925380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-0c7ee14f-a332-476d-bcb9-0d80a4d096d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:56.943850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7ee14f-a332-476d-bcb9-0d80a4d096d7 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:56.957890 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:56.958547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:56.958830 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:56.959110 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] PATH_INFO: `/v3/projects/9e57e45509e64231be32e5d652c57730` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:56.961773 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e57e45509e64231be32e5d652c57730)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:56.963769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:57.049832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e57e45509e64231be32e5d652c57730', 'request_id': 'req-2d75e5e5-2583-4121-8af3-74837b843aee'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.057124 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e57e45509e64231be32e5d652c57730', 'request_id': 'req-2d75e5e5-2583-4121-8af3-74837b843aee'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.060030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:57.068170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e57e45509e64231be32e5d652c57730', 'request_id': 'req-2d75e5e5-2583-4121-8af3-74837b843aee'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.084668 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] The token cache is being invalidate because project 9e57e45509e64231be32e5d652c57730 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:49:57.085013 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.086478 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.088572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1400/2767] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:49:56 2022] DELETE /identity/v3/projects/9e57e45509e64231be32e5d652c57730 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:57.093450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7748628-b360-4888-81de-f1c66c806f8a service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:57.105730 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7748628-b360-4888-81de-f1c66c806f8a service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:57.202911 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec65603-c015-4b5e-ba69-88d6240f3d9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:57.204846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec65603-c015-4b5e-ba69-88d6240f3d9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:57.205131 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec65603-c015-4b5e-ba69-88d6240f3d9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:57.205556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec65603-c015-4b5e-ba69-88d6240f3d9d admin admin] PATH_INFO: `/v3/users/0fa102bbaf6e446687157d8861e95561` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:57.208342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec65603-c015-4b5e-ba69-88d6240f3d9d admin admin] RBAC: Authorizing `identity:delete_user(user_id=0fa102bbaf6e446687157d8861e95561)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:57.213924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec65603-c015-4b5e-ba69-88d6240f3d9d admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:57.271774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-2ec65603-c015-4b5e-ba69-88d6240f3d9d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0fa102bbaf6e446687157d8861e95561'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.272758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-2ec65603-c015-4b5e-ba69-88d6240f3d9d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0fa102bbaf6e446687157d8861e95561'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.274661 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-2ec65603-c015-4b5e-ba69-88d6240f3d9d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0fa102bbaf6e446687157d8861e95561'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.283623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-2ec65603-c015-4b5e-ba69-88d6240f3d9d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0fa102bbaf6e446687157d8861e95561'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.297521 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-2ec65603-c015-4b5e-ba69-88d6240f3d9d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0fa102bbaf6e446687157d8861e95561'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.320694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1368/2768] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:49:57 2022] DELETE /identity/v3/users/0fa102bbaf6e446687157d8861e95561 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:49:57.698799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:57.701832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:57.719161 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d75e5e5-2583-4121-8af3-74837b843aee admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:57.787068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f4c8c2-c42b-4f99-b82f-1b53e6760f57 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:57.787363 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f4c8c2-c42b-4f99-b82f-1b53e6760f57 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:57.787650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f4c8c2-c42b-4f99-b82f-1b53e6760f57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:57.791525 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a4ee68-6ae4-4f94-af12-179db377ba6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:57.792247 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a4ee68-6ae4-4f94-af12-179db377ba6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:57.792689 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a4ee68-6ae4-4f94-af12-179db377ba6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:57.793035 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a4ee68-6ae4-4f94-af12-179db377ba6b admin admin] PATH_INFO: `/v3/projects/d84cb8858c304f87be0789a2afd3e211` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:57.794242 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a4ee68-6ae4-4f94-af12-179db377ba6b admin admin] RBAC: Authorizing `identity:delete_project(project_id=d84cb8858c304f87be0789a2afd3e211)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:57.797332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a4ee68-6ae4-4f94-af12-179db377ba6b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:57.818713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-d6f4c8c2-c42b-4f99-b82f-1b53e6760f57 None None] MFA Rules not processed for user `4d70d61439324d4b9c3b3513799b896c`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:57.820493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d6f4c8c2-c42b-4f99-b82f-1b53e6760f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:57.853567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-68a4ee68-6ae4-4f94-af12-179db377ba6b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd84cb8858c304f87be0789a2afd3e211', 'request_id': 'req-68a4ee68-6ae4-4f94-af12-179db377ba6b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.859384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-68a4ee68-6ae4-4f94-af12-179db377ba6b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd84cb8858c304f87be0789a2afd3e211', 'request_id': 'req-68a4ee68-6ae4-4f94-af12-179db377ba6b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.867464 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-68a4ee68-6ae4-4f94-af12-179db377ba6b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd84cb8858c304f87be0789a2afd3e211', 'request_id': 'req-68a4ee68-6ae4-4f94-af12-179db377ba6b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.873710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:57.880552 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-68a4ee68-6ae4-4f94-af12-179db377ba6b admin admin] The token cache is being invalidate because project d84cb8858c304f87be0789a2afd3e211 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:49:57.881183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-68a4ee68-6ae4-4f94-af12-179db377ba6b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.882174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-68a4ee68-6ae4-4f94-af12-179db377ba6b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:57.884979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1401/2769] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:49:57 2022] DELETE /identity/v3/projects/d84cb8858c304f87be0789a2afd3e211 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:57.890872 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00e7f9f-ba1e-47b2-9cb6-c6a32529fea8 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:57.891151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00e7f9f-ba1e-47b2-9cb6-c6a32529fea8 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:57.891650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00e7f9f-ba1e-47b2-9cb6-c6a32529fea8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:57.905813 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1369/2770] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:57 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:57.910868 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f4c8c2-c42b-4f99-b82f-1b53e6760f57 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:57.912372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d6f4c8c2-c42b-4f99-b82f-1b53e6760f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:57.920374 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-e00e7f9f-ba1e-47b2-9cb6-c6a32529fea8 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:57.921617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e00e7f9f-ba1e-47b2-9cb6-c6a32529fea8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:57.926807 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f4c8c2-c42b-4f99-b82f-1b53e6760f57 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:57.989197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941c4205-fcc3-4c65-a46b-66167376ebbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:57.989955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941c4205-fcc3-4c65-a46b-66167376ebbe admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:57.990324 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941c4205-fcc3-4c65-a46b-66167376ebbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:57.990665 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941c4205-fcc3-4c65-a46b-66167376ebbe admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:57.991237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1402/2771] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:57.992411 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941c4205-fcc3-4c65-a46b-66167376ebbe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:57.994370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941c4205-fcc3-4c65-a46b-66167376ebbe admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:58.000746 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9634640c-91be-42f5-a465-b1eee72f0ad8 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.001037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9634640c-91be-42f5-a465-b1eee72f0ad8 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.001314 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9634640c-91be-42f5-a465-b1eee72f0ad8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.022369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-9634640c-91be-42f5-a465-b1eee72f0ad8 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:58.023626 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-9634640c-91be-42f5-a465-b1eee72f0ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:58.026633 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1370/2772] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:49:57 2022] POST /identity/v3/projects => generated 377 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:49:58.030761 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941c4205-fcc3-4c65-a46b-66167376ebbe admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:58.042828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:58.109761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1403/2773] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:49:58.114012 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9634640c-91be-42f5-a465-b1eee72f0ad8 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:58.123195 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9634640c-91be-42f5-a465-b1eee72f0ad8 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:58.127953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836c6352-2b48-4465-bf68-3c0a3007bfc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:58.128166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941c4205-fcc3-4c65-a46b-66167376ebbe admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:58.128688 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836c6352-2b48-4465-bf68-3c0a3007bfc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.128887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836c6352-2b48-4465-bf68-3c0a3007bfc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.129110 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836c6352-2b48-4465-bf68-3c0a3007bfc3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.129751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836c6352-2b48-4465-bf68-3c0a3007bfc3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:49:58.130049 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836c6352-2b48-4465-bf68-3c0a3007bfc3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:58.131616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836c6352-2b48-4465-bf68-3c0a3007bfc3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:58.133061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b910ac90-6e93-4028-b93b-a63c420dbdc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:58.133654 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b910ac90-6e93-4028-b93b-a63c420dbdc9 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.133937 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b910ac90-6e93-4028-b93b-a63c420dbdc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.134212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b910ac90-6e93-4028-b93b-a63c420dbdc9 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.134881 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b910ac90-6e93-4028-b93b-a63c420dbdc9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:58.136901 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b910ac90-6e93-4028-b93b-a63c420dbdc9 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:58.149237 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1404/2774] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:49:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 28 12:49:58.154634 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836c6352-2b48-4465-bf68-3c0a3007bfc3 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:58.166432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836c6352-2b48-4465-bf68-3c0a3007bfc3 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:58.168347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1371/2775] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:49:58 2022] POST /identity/v3/users => generated 295 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:49:58.171718 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f01f388-e5ff-4200-88e9-2fb6358c3f0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:58.172363 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f01f388-e5ff-4200-88e9-2fb6358c3f0d admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.172515 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b910ac90-6e93-4028-b93b-a63c420dbdc9 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:58.172686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f01f388-e5ff-4200-88e9-2fb6358c3f0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.172921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f01f388-e5ff-4200-88e9-2fb6358c3f0d admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.173567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f01f388-e5ff-4200-88e9-2fb6358c3f0d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:58.175494 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f01f388-e5ff-4200-88e9-2fb6358c3f0d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:58.183556 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b910ac90-6e93-4028-b93b-a63c420dbdc9 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:58.189484 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9573834b-bcca-4f68-8c17-f2419ba35874 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:58.190098 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9573834b-bcca-4f68-8c17-f2419ba35874 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.190546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9573834b-bcca-4f68-8c17-f2419ba35874 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.190826 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9573834b-bcca-4f68-8c17-f2419ba35874 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.191481 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9573834b-bcca-4f68-8c17-f2419ba35874 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:58.193135 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9573834b-bcca-4f68-8c17-f2419ba35874 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:58.209723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1372/2776] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:58 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:49:58.209815 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1405/2777] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:49:58 2022] POST /identity/v3/projects => generated 387 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:49:58.212973 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9573834b-bcca-4f68-8c17-f2419ba35874 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:58.213460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f01f388-e5ff-4200-88e9-2fb6358c3f0d admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:58.229210 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9573834b-bcca-4f68-8c17-f2419ba35874 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:58.286815 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:58.287674 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f01f388-e5ff-4200-88e9-2fb6358c3f0d admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:58.308728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cccf1d-2682-47a3-8888-9e23c5275b88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:58.309453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cccf1d-2682-47a3-8888-9e23c5275b88 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.309832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cccf1d-2682-47a3-8888-9e23c5275b88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.309945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5620bb-8889-4eea-98ae-25a24c34537d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:58.310294 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cccf1d-2682-47a3-8888-9e23c5275b88 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.310524 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5620bb-8889-4eea-98ae-25a24c34537d admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.310812 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5620bb-8889-4eea-98ae-25a24c34537d admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.311086 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5620bb-8889-4eea-98ae-25a24c34537d admin admin] PATH_INFO: `/v3/projects/0fbb37c05a8d4c5bbeb0ba5adda9ac73/users/ca66846a50534121b1f997cdc85859f9/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.311194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52cccf1d-2682-47a3-8888-9e23c5275b88 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:58.313034 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52cccf1d-2682-47a3-8888-9e23c5275b88 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:58.328437 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5620bb-8889-4eea-98ae-25a24c34537d admin admin] RBAC: Authorizing `identity:create_grant(project_id=0fbb37c05a8d4c5bbeb0ba5adda9ac73, user_id=ca66846a50534121b1f997cdc85859f9, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:58.331353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5620bb-8889-4eea-98ae-25a24c34537d admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:58.347560 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1406/2778] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:49:58 2022] POST /identity/v3/users => generated 294 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:49:58.348827 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cccf1d-2682-47a3-8888-9e23c5275b88 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:58.352153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-52cccf1d-2682-47a3-8888-9e23c5275b88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:58.354509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1373/2779] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:58 2022] PUT /identity/v3/projects/0fbb37c05a8d4c5bbeb0ba5adda9ac73/users/ca66846a50534121b1f997cdc85859f9/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:49:58.355474 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5620bb-8889-4eea-98ae-25a24c34537d admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:58.369308 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cccf1d-2682-47a3-8888-9e23c5275b88 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:58.453172 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5f2ad2-a59f-4318-bc4d-f126255b4bf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:58.454481 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5f2ad2-a59f-4318-bc4d-f126255b4bf8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.454784 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5f2ad2-a59f-4318-bc4d-f126255b4bf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.455110 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5f2ad2-a59f-4318-bc4d-f126255b4bf8 admin admin] PATH_INFO: `/v3/users/34291935ced84832a77cbe0671ba4302` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.455216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5620bb-8889-4eea-98ae-25a24c34537d admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:58.457002 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5f2ad2-a59f-4318-bc4d-f126255b4bf8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=34291935ced84832a77cbe0671ba4302)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:58.459488 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5f2ad2-a59f-4318-bc4d-f126255b4bf8 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:58.462198 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e3a499-080a-4b07-84e9-f70e1426c6ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:58.465053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e3a499-080a-4b07-84e9-f70e1426c6ef admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.465591 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e3a499-080a-4b07-84e9-f70e1426c6ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.466816 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e3a499-080a-4b07-84e9-f70e1426c6ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.468331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e3a499-080a-4b07-84e9-f70e1426c6ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:58.471355 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e3a499-080a-4b07-84e9-f70e1426c6ef admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:58.488952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1374/2780] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:58 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:58.489679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e3a499-080a-4b07-84e9-f70e1426c6ef admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:58.492924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:58.502955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e3a499-080a-4b07-84e9-f70e1426c6ef admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:58.514407 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9dda5c5-d508-469d-8dc8-b190c050804b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:58.515616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dda5c5-d508-469d-8dc8-b190c050804b admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.516183 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dda5c5-d508-469d-8dc8-b190c050804b admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.516953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dda5c5-d508-469d-8dc8-b190c050804b admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.517431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9dda5c5-d508-469d-8dc8-b190c050804b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:58.519846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9dda5c5-d508-469d-8dc8-b190c050804b admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:58.533034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1375/2781] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:58 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:58.537078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9dda5c5-d508-469d-8dc8-b190c050804b admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:58.543344 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6e5f2ad2-a59f-4318-bc4d-f126255b4bf8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '34291935ced84832a77cbe0671ba4302'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:58.544146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6e5f2ad2-a59f-4318-bc4d-f126255b4bf8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '34291935ced84832a77cbe0671ba4302'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:58.548138 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9dda5c5-d508-469d-8dc8-b190c050804b admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:58.554023 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759e3e8d-0fe9-4816-b9b7-3eb7209bb54f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:58.555435 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759e3e8d-0fe9-4816-b9b7-3eb7209bb54f admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.555883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759e3e8d-0fe9-4816-b9b7-3eb7209bb54f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.556287 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759e3e8d-0fe9-4816-b9b7-3eb7209bb54f admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.557095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759e3e8d-0fe9-4816-b9b7-3eb7209bb54f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:58.559869 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759e3e8d-0fe9-4816-b9b7-3eb7209bb54f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:58.560682 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6e5f2ad2-a59f-4318-bc4d-f126255b4bf8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '34291935ced84832a77cbe0671ba4302'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:58.561801 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6e5f2ad2-a59f-4318-bc4d-f126255b4bf8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '34291935ced84832a77cbe0671ba4302'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:58.576595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1376/2782] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:58 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:58.577760 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759e3e8d-0fe9-4816-b9b7-3eb7209bb54f admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:58.579520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-759e3e8d-0fe9-4816-b9b7-3eb7209bb54f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:58.582684 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6e5f2ad2-a59f-4318-bc4d-f126255b4bf8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '34291935ced84832a77cbe0671ba4302'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:58.595974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759e3e8d-0fe9-4816-b9b7-3eb7209bb54f admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:58.602257 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f59d20-4fde-4d38-9d50-9688523dcefc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:58.603168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f59d20-4fde-4d38-9d50-9688523dcefc admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.603466 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f59d20-4fde-4d38-9d50-9688523dcefc admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.603740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f59d20-4fde-4d38-9d50-9688523dcefc admin admin] PATH_INFO: `/v3/projects/0fbb37c05a8d4c5bbeb0ba5adda9ac73/users/ca66846a50534121b1f997cdc85859f9/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.604966 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1407/2783] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:49:58 2022] DELETE /identity/v3/users/34291935ced84832a77cbe0671ba4302 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:58.606205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f59d20-4fde-4d38-9d50-9688523dcefc admin admin] RBAC: Authorizing `identity:create_grant(project_id=0fbb37c05a8d4c5bbeb0ba5adda9ac73, user_id=ca66846a50534121b1f997cdc85859f9, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:58.608115 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5f2ad2-a59f-4318-bc4d-f126255b4bf8 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:58.609213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f59d20-4fde-4d38-9d50-9688523dcefc admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:58.609546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6e5f2ad2-a59f-4318-bc4d-f126255b4bf8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:58.623112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1377/2784] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:58 2022] PUT /identity/v3/projects/0fbb37c05a8d4c5bbeb0ba5adda9ac73/users/ca66846a50534121b1f997cdc85859f9/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:58.624695 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772a08b1-6ac8-41fd-a417-c6b0217ad0de None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.624977 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772a08b1-6ac8-41fd-a417-c6b0217ad0de None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.625249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772a08b1-6ac8-41fd-a417-c6b0217ad0de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.649361 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-772a08b1-6ac8-41fd-a417-c6b0217ad0de None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:58.650670 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-772a08b1-6ac8-41fd-a417-c6b0217ad0de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:58.709597 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5f2ad2-a59f-4318-bc4d-f126255b4bf8 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:58.768124 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1378/2785] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:58.782848 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772a08b1-6ac8-41fd-a417-c6b0217ad0de None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:58.786182 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-772a08b1-6ac8-41fd-a417-c6b0217ad0de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:58.833134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baae2df6-1a4e-4787-920a-b80706d9a4b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:58.833993 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baae2df6-1a4e-4787-920a-b80706d9a4b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.834334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baae2df6-1a4e-4787-920a-b80706d9a4b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.834677 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baae2df6-1a4e-4787-920a-b80706d9a4b9 admin admin] PATH_INFO: `/v3/projects/2aa033f034844436b02c5b848a6e3636/users/065290fee44f4a27b3ecf879ba6f0207/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.851566 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:58.860409 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baae2df6-1a4e-4787-920a-b80706d9a4b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2aa033f034844436b02c5b848a6e3636, user_id=065290fee44f4a27b3ecf879ba6f0207, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:58.877637 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baae2df6-1a4e-4787-920a-b80706d9a4b9 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:58.891976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772a08b1-6ac8-41fd-a417-c6b0217ad0de None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:58.896545 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b905ce-5060-460d-9cdf-5111da2831fd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:58.897385 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b905ce-5060-460d-9cdf-5111da2831fd service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.898013 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b905ce-5060-460d-9cdf-5111da2831fd service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.898443 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b905ce-5060-460d-9cdf-5111da2831fd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.907583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1408/2786] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:58 2022] PUT /identity/v3/projects/2aa033f034844436b02c5b848a6e3636/users/065290fee44f4a27b3ecf879ba6f0207/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:58.911846 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b905ce-5060-460d-9cdf-5111da2831fd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:58.913680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b905ce-5060-460d-9cdf-5111da2831fd service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:58.919251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c998b81d-2f18-430a-9c7d-f0302c6cc553 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:58.920394 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c998b81d-2f18-430a-9c7d-f0302c6cc553 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:58.920856 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c998b81d-2f18-430a-9c7d-f0302c6cc553 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:58.927249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1379/2787] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:49:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:49:58.974900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-c998b81d-2f18-430a-9c7d-f0302c6cc553 None None] MFA Rules not processed for user `065290fee44f4a27b3ecf879ba6f0207`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:58.975910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c998b81d-2f18-430a-9c7d-f0302c6cc553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:58.998613 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:59.077121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1409/2788] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:58 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:59.081484 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b905ce-5060-460d-9cdf-5111da2831fd service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:59.095943 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b905ce-5060-460d-9cdf-5111da2831fd service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:59.197444 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899cf26e-83a9-4ebf-ab6f-31c98a00154b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:59.199454 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899cf26e-83a9-4ebf-ab6f-31c98a00154b admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:59.199963 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899cf26e-83a9-4ebf-ab6f-31c98a00154b admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:59.200833 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899cf26e-83a9-4ebf-ab6f-31c98a00154b admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:59.201711 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899cf26e-83a9-4ebf-ab6f-31c98a00154b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:59.204326 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899cf26e-83a9-4ebf-ab6f-31c98a00154b admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:59.253066 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1380/2789] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:49:59 2022] POST /identity/v3/projects => generated 385 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:49:59.253298 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c998b81d-2f18-430a-9c7d-f0302c6cc553 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:59.265169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c998b81d-2f18-430a-9c7d-f0302c6cc553 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:59.334402 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899cf26e-83a9-4ebf-ab6f-31c98a00154b admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:59.336317 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-899cf26e-83a9-4ebf-ab6f-31c98a00154b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:59.355083 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8c52a6-262d-41c7-80ac-53b8cfc5d667 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:59.355905 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8c52a6-262d-41c7-80ac-53b8cfc5d667 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:59.356256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8c52a6-262d-41c7-80ac-53b8cfc5d667 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:59.356643 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8c52a6-262d-41c7-80ac-53b8cfc5d667 admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:59.357511 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8c52a6-262d-41c7-80ac-53b8cfc5d667 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:59.359515 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8c52a6-262d-41c7-80ac-53b8cfc5d667 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:59.410519 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1410/2790] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:49:59 2022] POST /identity/v3/users => generated 293 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:49:59.417832 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8c52a6-262d-41c7-80ac-53b8cfc5d667 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:59.428564 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8c52a6-262d-41c7-80ac-53b8cfc5d667 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:59.434747 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f71008e-0b58-49b6-812e-c2860629c86b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:59.436041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f71008e-0b58-49b6-812e-c2860629c86b admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:59.436041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f71008e-0b58-49b6-812e-c2860629c86b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:59.436366 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f71008e-0b58-49b6-812e-c2860629c86b admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:59.437135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f71008e-0b58-49b6-812e-c2860629c86b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:59.438808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f71008e-0b58-49b6-812e-c2860629c86b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:59.450713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1411/2791] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:59 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:49:59.454685 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f71008e-0b58-49b6-812e-c2860629c86b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:59.485112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f71008e-0b58-49b6-812e-c2860629c86b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:59.497823 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52d872d-0a50-4690-acbc-c68f26c5a45d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:59.498447 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d872d-0a50-4690-acbc-c68f26c5a45d admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:59.498729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d872d-0a50-4690-acbc-c68f26c5a45d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:59.499006 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d872d-0a50-4690-acbc-c68f26c5a45d admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:59.499663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52d872d-0a50-4690-acbc-c68f26c5a45d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:59.501302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52d872d-0a50-4690-acbc-c68f26c5a45d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:59.532514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899cf26e-83a9-4ebf-ab6f-31c98a00154b admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:59.535709 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1412/2792] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:49:59 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:49:59.537516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52d872d-0a50-4690-acbc-c68f26c5a45d admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:59.546453 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:59.547600 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:59.547906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:59.548189 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] PATH_INFO: `/v3/projects/81a954e41aee492390d703b330653728` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:59.549797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=81a954e41aee492390d703b330653728)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:59.554012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:59.557146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52d872d-0a50-4690-acbc-c68f26c5a45d admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:49:59.566741 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e52f2f0-45d8-49f1-97de-402c41647c24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:49:59.568079 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e52f2f0-45d8-49f1-97de-402c41647c24 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:59.568735 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e52f2f0-45d8-49f1-97de-402c41647c24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:59.569200 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e52f2f0-45d8-49f1-97de-402c41647c24 admin admin] PATH_INFO: `/v3/projects/a6940ce57b3240fdb472ab88e889355b/users/3ea91d0ebb9f4a22ac391b7d52ecf555/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:59.620454 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e52f2f0-45d8-49f1-97de-402c41647c24 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6940ce57b3240fdb472ab88e889355b, user_id=3ea91d0ebb9f4a22ac391b7d52ecf555, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:49:59.624773 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e52f2f0-45d8-49f1-97de-402c41647c24 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:49:59.643970 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1413/2793] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:49:59 2022] PUT /identity/v3/projects/a6940ce57b3240fdb472ab88e889355b/users/3ea91d0ebb9f4a22ac391b7d52ecf555/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:59.658998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43a039a-7b2c-4a35-bd0f-5fa4cfd4556a None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:49:59.659547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43a039a-7b2c-4a35-bd0f-5fa4cfd4556a None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:49:59.659892 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43a039a-7b2c-4a35-bd0f-5fa4cfd4556a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:49:59.685452 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81a954e41aee492390d703b330653728', 'request_id': 'req-76da6a0b-2d75-4961-af1d-109941f42be4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:59.692717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81a954e41aee492390d703b330653728', 'request_id': 'req-76da6a0b-2d75-4961-af1d-109941f42be4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:59.698482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:59.706080 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81a954e41aee492390d703b330653728', 'request_id': 'req-76da6a0b-2d75-4961-af1d-109941f42be4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:59.723744 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-a43a039a-7b2c-4a35-bd0f-5fa4cfd4556a None None] MFA Rules not processed for user `3ea91d0ebb9f4a22ac391b7d52ecf555`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:49:59.725969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a43a039a-7b2c-4a35-bd0f-5fa4cfd4556a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:49:59.735674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] The token cache is being invalidate because project 81a954e41aee492390d703b330653728 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:49:59.736027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:59.739265 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:49:59.765014 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1381/2794] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:49:59 2022] DELETE /identity/v3/projects/81a954e41aee492390d703b330653728 => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:49:59.856147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:49:59.904128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1414/2795] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:49:59 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:49:59.909848 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:49:59.914837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:00.029796 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76da6a0b-2d75-4961-af1d-109941f42be4 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:00.034925 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb9d934-fc1e-426d-a4ae-6c4b0afa89b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:00.035603 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb9d934-fc1e-426d-a4ae-6c4b0afa89b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:00.035890 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb9d934-fc1e-426d-a4ae-6c4b0afa89b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:00.036165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb9d934-fc1e-426d-a4ae-6c4b0afa89b7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:00.036872 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb9d934-fc1e-426d-a4ae-6c4b0afa89b7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:00.039033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb9d934-fc1e-426d-a4ae-6c4b0afa89b7 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:00.046711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43a039a-7b2c-4a35-bd0f-5fa4cfd4556a None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:00.048888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a43a039a-7b2c-4a35-bd0f-5fa4cfd4556a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:00.085530 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1382/2796] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:49:59 2022] POST /identity/v3/projects => generated 385 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:00.088804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb9d934-fc1e-426d-a4ae-6c4b0afa89b7 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:00.125274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43a039a-7b2c-4a35-bd0f-5fa4cfd4556a None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:00.189033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb9d934-fc1e-426d-a4ae-6c4b0afa89b7 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:00.190647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1624b12a-36f1-454f-a2ac-3ed931d247ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:00.191617 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1624b12a-36f1-454f-a2ac-3ed931d247ab service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:00.191782 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1624b12a-36f1-454f-a2ac-3ed931d247ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:00.192286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1624b12a-36f1-454f-a2ac-3ed931d247ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:00.194277 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1624b12a-36f1-454f-a2ac-3ed931d247ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:00.195424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5666f3-58be-4066-afa9-845b04af26b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:00.196882 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5666f3-58be-4066-afa9-845b04af26b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:00.197261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5666f3-58be-4066-afa9-845b04af26b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:00.197989 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5666f3-58be-4066-afa9-845b04af26b8 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:00.198565 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5666f3-58be-4066-afa9-845b04af26b8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:00.200483 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5666f3-58be-4066-afa9-845b04af26b8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:00.215285 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1624b12a-36f1-454f-a2ac-3ed931d247ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:00.217680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1624b12a-36f1-454f-a2ac-3ed931d247ab service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:00.230108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1383/2797] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:50:00 2022] POST /identity/v3/users => generated 299 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:00.230342 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1415/2798] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:00.231998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5666f3-58be-4066-afa9-845b04af26b8 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:00.244914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5666f3-58be-4066-afa9-845b04af26b8 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:00.250548 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d32176b-c5f4-4b3e-ba8f-6eed9d6f15b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:00.252412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d32176b-c5f4-4b3e-ba8f-6eed9d6f15b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:00.252987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d32176b-c5f4-4b3e-ba8f-6eed9d6f15b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:00.253546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d32176b-c5f4-4b3e-ba8f-6eed9d6f15b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:00.254499 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d32176b-c5f4-4b3e-ba8f-6eed9d6f15b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:00.256412 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d32176b-c5f4-4b3e-ba8f-6eed9d6f15b9 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:00.276505 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1384/2799] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:00 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:00.277950 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1624b12a-36f1-454f-a2ac-3ed931d247ab service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:00.300777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1624b12a-36f1-454f-a2ac-3ed931d247ab service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:00.315068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415ca9a2-ab64-43ce-b505-46f202e767e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:00.324132 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415ca9a2-ab64-43ce-b505-46f202e767e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:00.324475 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415ca9a2-ab64-43ce-b505-46f202e767e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:00.324760 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415ca9a2-ab64-43ce-b505-46f202e767e0 admin admin] PATH_INFO: `/v3/projects/8121ab3abc5f429083ee5d708df2bd97/users/3c3c7c11230349a5b3f846658881146a/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:00.357605 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415ca9a2-ab64-43ce-b505-46f202e767e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8121ab3abc5f429083ee5d708df2bd97, user_id=3c3c7c11230349a5b3f846658881146a, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:00.360740 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415ca9a2-ab64-43ce-b505-46f202e767e0 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:00.386895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1416/2800] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:50:00 2022] PUT /identity/v3/projects/8121ab3abc5f429083ee5d708df2bd97/users/3c3c7c11230349a5b3f846658881146a/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:50:00.390679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d32176b-c5f4-4b3e-ba8f-6eed9d6f15b9 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:00.404257 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d32176b-c5f4-4b3e-ba8f-6eed9d6f15b9 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:00.502481 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:00.521708 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853b10f4-b61c-437a-ba66-2a2895ef20ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:00.522692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853b10f4-b61c-437a-ba66-2a2895ef20ce admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:00.523418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853b10f4-b61c-437a-ba66-2a2895ef20ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:00.523909 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853b10f4-b61c-437a-ba66-2a2895ef20ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:00.524783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853b10f4-b61c-437a-ba66-2a2895ef20ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:00.526597 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853b10f4-b61c-437a-ba66-2a2895ef20ce admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:00.539400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1385/2801] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:00 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:00.544932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415ca9a2-ab64-43ce-b505-46f202e767e0 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:00.565294 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415ca9a2-ab64-43ce-b505-46f202e767e0 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:00.573203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d30c7f-a9da-4888-8d79-c947876de17b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:00.574468 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d30c7f-a9da-4888-8d79-c947876de17b admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:00.574752 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d30c7f-a9da-4888-8d79-c947876de17b admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:00.575390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d30c7f-a9da-4888-8d79-c947876de17b admin admin] PATH_INFO: `/v3/projects/8121ab3abc5f429083ee5d708df2bd97/users/3c3c7c11230349a5b3f846658881146a/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:00.577621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d30c7f-a9da-4888-8d79-c947876de17b admin admin] RBAC: Authorizing `identity:create_grant(project_id=8121ab3abc5f429083ee5d708df2bd97, user_id=3c3c7c11230349a5b3f846658881146a, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:00.580275 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d30c7f-a9da-4888-8d79-c947876de17b admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:00.598063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1417/2802] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:50:00 2022] PUT /identity/v3/projects/8121ab3abc5f429083ee5d708df2bd97/users/3c3c7c11230349a5b3f846658881146a/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:00.614605 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352e5fbf-3719-48a2-8a06-bf6e236825c7 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:00.615285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352e5fbf-3719-48a2-8a06-bf6e236825c7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:00.617300 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352e5fbf-3719-48a2-8a06-bf6e236825c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:00.677926 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-352e5fbf-3719-48a2-8a06-bf6e236825c7 None None] MFA Rules not processed for user `3c3c7c11230349a5b3f846658881146a`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:00.680652 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-352e5fbf-3719-48a2-8a06-bf6e236825c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:00.792573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1386/2803] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:00 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:00.817649 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d30c7f-a9da-4888-8d79-c947876de17b admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:00.821647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-46d30c7f-a9da-4888-8d79-c947876de17b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:00.944618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d30c7f-a9da-4888-8d79-c947876de17b admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:00.950355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68d2415-01c3-409f-b731-06cd5cac9562 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:00.951358 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68d2415-01c3-409f-b731-06cd5cac9562 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:00.951953 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68d2415-01c3-409f-b731-06cd5cac9562 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:00.952548 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68d2415-01c3-409f-b731-06cd5cac9562 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:00.970181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68d2415-01c3-409f-b731-06cd5cac9562 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:00.971921 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68d2415-01c3-409f-b731-06cd5cac9562 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:00.991174 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1418/2804] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:00 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:01.077904 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352e5fbf-3719-48a2-8a06-bf6e236825c7 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:01.079562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-352e5fbf-3719-48a2-8a06-bf6e236825c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:01.252863 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352e5fbf-3719-48a2-8a06-bf6e236825c7 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:01.260611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f697c0b-67d2-43c2-8f85-db305d8a9246 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:01.261755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f697c0b-67d2-43c2-8f85-db305d8a9246 service placement] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:01.262187 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f697c0b-67d2-43c2-8f85-db305d8a9246 service placement] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:01.262618 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f697c0b-67d2-43c2-8f85-db305d8a9246 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:01.264700 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5f697c0b-67d2-43c2-8f85-db305d8a9246 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:01.282151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f697c0b-67d2-43c2-8f85-db305d8a9246 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:01.284783 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f697c0b-67d2-43c2-8f85-db305d8a9246 service placement] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:01.312737 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1387/2805] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:01 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:02.013640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68d2415-01c3-409f-b731-06cd5cac9562 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:02.016563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a68d2415-01c3-409f-b731-06cd5cac9562 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:02.037389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68d2415-01c3-409f-b731-06cd5cac9562 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:02.044664 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2744af08-c3ee-4cd0-b2ab-7e01b51025dc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:02.045720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2744af08-c3ee-4cd0-b2ab-7e01b51025dc service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:02.046233 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2744af08-c3ee-4cd0-b2ab-7e01b51025dc service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:02.046982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2744af08-c3ee-4cd0-b2ab-7e01b51025dc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:02.050309 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2744af08-c3ee-4cd0-b2ab-7e01b51025dc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:02.064484 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:02.145219 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2744af08-c3ee-4cd0-b2ab-7e01b51025dc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:02.147316 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2744af08-c3ee-4cd0-b2ab-7e01b51025dc service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:02.191660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1419/2806] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:02 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:04.187818 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f697c0b-67d2-43c2-8f85-db305d8a9246 service placement] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:04.189431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5f697c0b-67d2-43c2-8f85-db305d8a9246 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:04.355915 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f697c0b-67d2-43c2-8f85-db305d8a9246 service placement] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:04.369031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3248ef0-d60e-42c7-9889-8cb4a4731e56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:04.369851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3248ef0-d60e-42c7-9889-8cb4a4731e56 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:04.370304 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3248ef0-d60e-42c7-9889-8cb4a4731e56 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:04.370576 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3248ef0-d60e-42c7-9889-8cb4a4731e56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:04.376181 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d3248ef0-d60e-42c7-9889-8cb4a4731e56 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:04.405998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3248ef0-d60e-42c7-9889-8cb4a4731e56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:04.421471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3248ef0-d60e-42c7-9889-8cb4a4731e56 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:04.450498 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1388/2807] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:04 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:05.168073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050c838b-a92e-4217-8731-de7bd6fe39ff None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:05.169080 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050c838b-a92e-4217-8731-de7bd6fe39ff None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:05.169508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050c838b-a92e-4217-8731-de7bd6fe39ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:05.200990 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-050c838b-a92e-4217-8731-de7bd6fe39ff None None] MFA Rules not processed for user `ca66846a50534121b1f997cdc85859f9`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:05.202379 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-050c838b-a92e-4217-8731-de7bd6fe39ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:05.296574 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1420/2808] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:05 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:05.305306 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3248ef0-d60e-42c7-9889-8cb4a4731e56 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:05.307393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d3248ef0-d60e-42c7-9889-8cb4a4731e56 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:05.319239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=100 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:05.339226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050c838b-a92e-4217-8731-de7bd6fe39ff None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:05.342146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-050c838b-a92e-4217-8731-de7bd6fe39ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:05.439616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3248ef0-d60e-42c7-9889-8cb4a4731e56 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:05.450589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ca4b98-622c-46bd-990a-87e6783b5b33 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:05.451500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ca4b98-622c-46bd-990a-87e6783b5b33 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:05.451785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ca4b98-622c-46bd-990a-87e6783b5b33 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:05.452090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ca4b98-622c-46bd-990a-87e6783b5b33 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:05.458162 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-21ca4b98-622c-46bd-990a-87e6783b5b33 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:05.477636 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050c838b-a92e-4217-8731-de7bd6fe39ff None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:05.484758 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff4c584-cf96-426f-8a97-6d621c31ad55 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:05.485532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff4c584-cf96-426f-8a97-6d621c31ad55 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:05.485910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff4c584-cf96-426f-8a97-6d621c31ad55 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:05.486280 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff4c584-cf96-426f-8a97-6d621c31ad55 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:05.493578 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-3ff4c584-cf96-426f-8a97-6d621c31ad55 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:05.509076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff4c584-cf96-426f-8a97-6d621c31ad55 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:05.511210 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff4c584-cf96-426f-8a97-6d621c31ad55 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:05.530119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ca4b98-622c-46bd-990a-87e6783b5b33 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:05.531766 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ca4b98-622c-46bd-990a-87e6783b5b33 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:05.542112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1421/2809] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:05.553428 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1389/2810] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:05 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:07.123224 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff4c584-cf96-426f-8a97-6d621c31ad55 service nova] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:07.136627 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff4c584-cf96-426f-8a97-6d621c31ad55 service nova] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:07.140728 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f98ce4-3915-406d-a41c-ee9ff0b1cfba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:07.141597 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f98ce4-3915-406d-a41c-ee9ff0b1cfba service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:07.141831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f98ce4-3915-406d-a41c-ee9ff0b1cfba service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:07.142297 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f98ce4-3915-406d-a41c-ee9ff0b1cfba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:07.163651 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:07.229600 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f98ce4-3915-406d-a41c-ee9ff0b1cfba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:07.231876 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f98ce4-3915-406d-a41c-ee9ff0b1cfba service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:07.249527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1422/2811] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:07 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:09.683729 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ca4b98-622c-46bd-990a-87e6783b5b33 service nova] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:09.758451 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ca4b98-622c-46bd-990a-87e6783b5b33 service nova] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:09.764191 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d5edb8-848c-4d22-a5ed-289009816866 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:09.764899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d5edb8-848c-4d22-a5ed-289009816866 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:09.765157 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d5edb8-848c-4d22-a5ed-289009816866 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:09.765444 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d5edb8-848c-4d22-a5ed-289009816866 admin admin] PATH_INFO: `/v3/users/065290fee44f4a27b3ecf879ba6f0207` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:09.766602 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d5edb8-848c-4d22-a5ed-289009816866 admin admin] RBAC: Authorizing `identity:delete_user(user_id=065290fee44f4a27b3ecf879ba6f0207)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:09.768677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d5edb8-848c-4d22-a5ed-289009816866 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:09.843154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-39d5edb8-848c-4d22-a5ed-289009816866 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '065290fee44f4a27b3ecf879ba6f0207'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:09.844097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-39d5edb8-848c-4d22-a5ed-289009816866 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '065290fee44f4a27b3ecf879ba6f0207'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:09.844842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-39d5edb8-848c-4d22-a5ed-289009816866 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '065290fee44f4a27b3ecf879ba6f0207'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:09.846398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:09.855272 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-39d5edb8-848c-4d22-a5ed-289009816866 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '065290fee44f4a27b3ecf879ba6f0207'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:09.868636 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-39d5edb8-848c-4d22-a5ed-289009816866 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '065290fee44f4a27b3ecf879ba6f0207'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:09.886002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1390/2812] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:50:09 2022] DELETE /identity/v3/users/065290fee44f4a27b3ecf879ba6f0207 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:09.890684 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05ff0c7-e4c5-425a-91d5-3c21a92b621b None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:09.891169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05ff0c7-e4c5-425a-91d5-3c21a92b621b None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:09.891262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05ff0c7-e4c5-425a-91d5-3c21a92b621b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:09.910539 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-f05ff0c7-e4c5-425a-91d5-3c21a92b621b None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:09.911797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f05ff0c7-e4c5-425a-91d5-3c21a92b621b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:09.974351 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1423/2813] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:09.984785 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d5edb8-848c-4d22-a5ed-289009816866 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:09.986322 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-39d5edb8-848c-4d22-a5ed-289009816866 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:10.044591 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d5edb8-848c-4d22-a5ed-289009816866 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:10.049090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a71b5d-c6ae-4e2d-adb5-5b5cfb1e1e41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:10.050717 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a71b5d-c6ae-4e2d-adb5-5b5cfb1e1e41 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:10.051002 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a71b5d-c6ae-4e2d-adb5-5b5cfb1e1e41 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:10.051275 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a71b5d-c6ae-4e2d-adb5-5b5cfb1e1e41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:10.060900 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a71b5d-c6ae-4e2d-adb5-5b5cfb1e1e41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:10.062335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a71b5d-c6ae-4e2d-adb5-5b5cfb1e1e41 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:10.071605 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1391/2814] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:10.350137 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05ff0c7-e4c5-425a-91d5-3c21a92b621b None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:10.352026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-f05ff0c7-e4c5-425a-91d5-3c21a92b621b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:10.366160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05ff0c7-e4c5-425a-91d5-3c21a92b621b None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:10.371436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:10.372289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:10.372891 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:10.373320 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] PATH_INFO: `/v3/projects/2aa033f034844436b02c5b848a6e3636` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:10.374625 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2aa033f034844436b02c5b848a6e3636)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:10.376713 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:10.434976 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2aa033f034844436b02c5b848a6e3636', 'request_id': 'req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:10.441625 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2aa033f034844436b02c5b848a6e3636', 'request_id': 'req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:10.451179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2aa033f034844436b02c5b848a6e3636', 'request_id': 'req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:10.464266 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] The token cache is being invalidate because project 2aa033f034844436b02c5b848a6e3636 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:50:10.464589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:10.465234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:10.466720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1424/2815] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:50:10 2022] DELETE /identity/v3/projects/2aa033f034844436b02c5b848a6e3636 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:10.470898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a71b5d-c6ae-4e2d-adb5-5b5cfb1e1e41 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:10.490902 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a71b5d-c6ae-4e2d-adb5-5b5cfb1e1e41 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:10.560284 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d5f88b-36bb-4c9f-b4f3-4b3ab9c4012a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:10.560945 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d5f88b-36bb-4c9f-b4f3-4b3ab9c4012a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:10.561224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d5f88b-36bb-4c9f-b4f3-4b3ab9c4012a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:10.561496 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d5f88b-36bb-4c9f-b4f3-4b3ab9c4012a admin admin] PATH_INFO: `/v3/users/3ea91d0ebb9f4a22ac391b7d52ecf555` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:10.562609 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d5f88b-36bb-4c9f-b4f3-4b3ab9c4012a admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ea91d0ebb9f4a22ac391b7d52ecf555)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:10.564621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d5f88b-36bb-4c9f-b4f3-4b3ab9c4012a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:10.617616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-53d5f88b-36bb-4c9f-b4f3-4b3ab9c4012a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ea91d0ebb9f4a22ac391b7d52ecf555'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:10.618276 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-53d5f88b-36bb-4c9f-b4f3-4b3ab9c4012a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ea91d0ebb9f4a22ac391b7d52ecf555'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:10.618889 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-53d5f88b-36bb-4c9f-b4f3-4b3ab9c4012a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ea91d0ebb9f4a22ac391b7d52ecf555'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:10.628530 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-53d5f88b-36bb-4c9f-b4f3-4b3ab9c4012a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ea91d0ebb9f4a22ac391b7d52ecf555'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:10.642878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:10.645630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-53d5f88b-36bb-4c9f-b4f3-4b3ab9c4012a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ea91d0ebb9f4a22ac391b7d52ecf555'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:10.660526 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1392/2816] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:50:10 2022] DELETE /identity/v3/users/3ea91d0ebb9f4a22ac391b7d52ecf555 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:11.031012 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:11.032528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:11.046535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fca599-ff0f-491e-86aa-0f4ebcfb5fb8 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:11.093475 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:11.110799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:11.111425 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:11.111696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:11.111968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] PATH_INFO: `/v3/projects/a6940ce57b3240fdb472ab88e889355b` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:11.113421 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a6940ce57b3240fdb472ab88e889355b)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:11.115307 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:11.162439 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a6940ce57b3240fdb472ab88e889355b', 'request_id': 'req-02b587c7-bdad-4539-b9ae-e08e4be08360'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.168066 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a6940ce57b3240fdb472ab88e889355b', 'request_id': 'req-02b587c7-bdad-4539-b9ae-e08e4be08360'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.175653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a6940ce57b3240fdb472ab88e889355b', 'request_id': 'req-02b587c7-bdad-4539-b9ae-e08e4be08360'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.188146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] The token cache is being invalidate because project a6940ce57b3240fdb472ab88e889355b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:50:11.188467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.189117 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.190598 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1425/2817] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:50:11 2022] DELETE /identity/v3/projects/a6940ce57b3240fdb472ab88e889355b => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:11.194672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d5f88b-36bb-4c9f-b4f3-4b3ab9c4012a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:11.196650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-53d5f88b-36bb-4c9f-b4f3-4b3ab9c4012a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:11.209054 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d5f88b-36bb-4c9f-b4f3-4b3ab9c4012a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:11.265689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391bbff8-7604-45fc-9de6-f959181d5a45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:11.266308 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391bbff8-7604-45fc-9de6-f959181d5a45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:11.266586 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391bbff8-7604-45fc-9de6-f959181d5a45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:11.266859 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391bbff8-7604-45fc-9de6-f959181d5a45 admin admin] PATH_INFO: `/v3/users/3c3c7c11230349a5b3f846658881146a` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:11.267973 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391bbff8-7604-45fc-9de6-f959181d5a45 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c3c7c11230349a5b3f846658881146a)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:11.269930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391bbff8-7604-45fc-9de6-f959181d5a45 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:11.316066 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-391bbff8-7604-45fc-9de6-f959181d5a45 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c3c7c11230349a5b3f846658881146a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.316643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-391bbff8-7604-45fc-9de6-f959181d5a45 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c3c7c11230349a5b3f846658881146a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.317240 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-391bbff8-7604-45fc-9de6-f959181d5a45 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c3c7c11230349a5b3f846658881146a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.324740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-391bbff8-7604-45fc-9de6-f959181d5a45 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c3c7c11230349a5b3f846658881146a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.339541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-391bbff8-7604-45fc-9de6-f959181d5a45 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c3c7c11230349a5b3f846658881146a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.352100 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1393/2818] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:50:11 2022] DELETE /identity/v3/users/3c3c7c11230349a5b3f846658881146a => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:11.637447 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:11.639336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:11.653557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b587c7-bdad-4539-b9ae-e08e4be08360 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:11.712137 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b0e578-af12-4c5c-bf22-b5064f0a517d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:11.712819 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b0e578-af12-4c5c-bf22-b5064f0a517d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:11.713170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b0e578-af12-4c5c-bf22-b5064f0a517d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:11.713444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b0e578-af12-4c5c-bf22-b5064f0a517d admin admin] PATH_INFO: `/v3/projects/8121ab3abc5f429083ee5d708df2bd97` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:11.714571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b0e578-af12-4c5c-bf22-b5064f0a517d admin admin] RBAC: Authorizing `identity:delete_project(project_id=8121ab3abc5f429083ee5d708df2bd97)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:11.716502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b0e578-af12-4c5c-bf22-b5064f0a517d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:11.763739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-54b0e578-af12-4c5c-bf22-b5064f0a517d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8121ab3abc5f429083ee5d708df2bd97', 'request_id': 'req-54b0e578-af12-4c5c-bf22-b5064f0a517d'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.769289 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-54b0e578-af12-4c5c-bf22-b5064f0a517d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8121ab3abc5f429083ee5d708df2bd97', 'request_id': 'req-54b0e578-af12-4c5c-bf22-b5064f0a517d'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.776606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-54b0e578-af12-4c5c-bf22-b5064f0a517d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8121ab3abc5f429083ee5d708df2bd97', 'request_id': 'req-54b0e578-af12-4c5c-bf22-b5064f0a517d'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.789060 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-54b0e578-af12-4c5c-bf22-b5064f0a517d admin admin] The token cache is being invalidate because project 8121ab3abc5f429083ee5d708df2bd97 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:50:11.789355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-54b0e578-af12-4c5c-bf22-b5064f0a517d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.790026 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-54b0e578-af12-4c5c-bf22-b5064f0a517d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:11.791566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1426/2819] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:50:11 2022] DELETE /identity/v3/projects/8121ab3abc5f429083ee5d708df2bd97 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:11.797518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969544ed-afbb-4784-8330-d96efb84979e None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:11.797806 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969544ed-afbb-4784-8330-d96efb84979e None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:11.798085 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969544ed-afbb-4784-8330-d96efb84979e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:11.817720 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-969544ed-afbb-4784-8330-d96efb84979e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:11.818917 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-969544ed-afbb-4784-8330-d96efb84979e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:11.882689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1394/2820] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:11.890770 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4975b7b-4d91-457c-b90d-038dfac2412e None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:11.891075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4975b7b-4d91-457c-b90d-038dfac2412e None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:11.891334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4975b7b-4d91-457c-b90d-038dfac2412e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:11.899518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:11.916053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-a4975b7b-4d91-457c-b90d-038dfac2412e None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:11.917349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-a4975b7b-4d91-457c-b90d-038dfac2412e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:11.925404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1427/2821] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:11.929657 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969544ed-afbb-4784-8330-d96efb84979e None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:11.939720 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969544ed-afbb-4784-8330-d96efb84979e None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:11.944595 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62232bc6-6889-4a8d-b178-8d14cc494fa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:11.945204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62232bc6-6889-4a8d-b178-8d14cc494fa8 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:11.945482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62232bc6-6889-4a8d-b178-8d14cc494fa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:11.945753 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62232bc6-6889-4a8d-b178-8d14cc494fa8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:11.946404 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62232bc6-6889-4a8d-b178-8d14cc494fa8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:50:11.946709 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62232bc6-6889-4a8d-b178-8d14cc494fa8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:11.948340 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62232bc6-6889-4a8d-b178-8d14cc494fa8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:11.949974 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:11.963920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1395/2822] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:50:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:50:11.967452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4975b7b-4d91-457c-b90d-038dfac2412e None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:11.977516 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4975b7b-4d91-457c-b90d-038dfac2412e None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:11.982244 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e510497f-49b6-4a14-a9df-ecea7e37da65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:11.982839 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e510497f-49b6-4a14-a9df-ecea7e37da65 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:11.983114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e510497f-49b6-4a14-a9df-ecea7e37da65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:11.983382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e510497f-49b6-4a14-a9df-ecea7e37da65 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:11.984052 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e510497f-49b6-4a14-a9df-ecea7e37da65 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:11.986012 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e510497f-49b6-4a14-a9df-ecea7e37da65 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:12.015286 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1428/2823] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:50:11 2022] POST /identity/v3/projects => generated 399 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:12.019699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62232bc6-6889-4a8d-b178-8d14cc494fa8 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:12.032847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62232bc6-6889-4a8d-b178-8d14cc494fa8 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:12.092467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0a877c-f728-4f6c-a40e-eddc269f168d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:12.093243 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0a877c-f728-4f6c-a40e-eddc269f168d admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:12.093668 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0a877c-f728-4f6c-a40e-eddc269f168d admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:12.093892 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0a877c-f728-4f6c-a40e-eddc269f168d admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:12.094562 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0a877c-f728-4f6c-a40e-eddc269f168d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:12.096612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0a877c-f728-4f6c-a40e-eddc269f168d admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:12.121627 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1396/2824] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:50:12 2022] POST /identity/v3/users => generated 300 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:50:12.125967 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e510497f-49b6-4a14-a9df-ecea7e37da65 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:12.138722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e510497f-49b6-4a14-a9df-ecea7e37da65 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:12.143441 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffe9a11-a48b-4a79-b5de-ea092a64e47e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:12.144076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffe9a11-a48b-4a79-b5de-ea092a64e47e admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:12.144391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffe9a11-a48b-4a79-b5de-ea092a64e47e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:12.144666 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffe9a11-a48b-4a79-b5de-ea092a64e47e admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:12.145371 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffe9a11-a48b-4a79-b5de-ea092a64e47e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:12.146956 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffe9a11-a48b-4a79-b5de-ea092a64e47e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:12.158160 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1429/2825] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:12 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:12.162953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0a877c-f728-4f6c-a40e-eddc269f168d admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:12.174197 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0a877c-f728-4f6c-a40e-eddc269f168d admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:12.179395 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afda29dd-9686-40b3-9795-c097175fa820 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:12.180064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afda29dd-9686-40b3-9795-c097175fa820 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:12.180443 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afda29dd-9686-40b3-9795-c097175fa820 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:12.180862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afda29dd-9686-40b3-9795-c097175fa820 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:12.181502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afda29dd-9686-40b3-9795-c097175fa820 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:12.183103 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afda29dd-9686-40b3-9795-c097175fa820 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:12.194235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1397/2826] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:12 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:12.201567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffe9a11-a48b-4a79-b5de-ea092a64e47e admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:12.214498 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffe9a11-a48b-4a79-b5de-ea092a64e47e admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:12.219257 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84d3224-a163-4d83-b835-055e51b09b27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:12.219942 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84d3224-a163-4d83-b835-055e51b09b27 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:12.220221 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84d3224-a163-4d83-b835-055e51b09b27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:12.220528 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84d3224-a163-4d83-b835-055e51b09b27 admin admin] PATH_INFO: `/v3/projects/6b039a350343464abc724850d42ace5c/users/130a6da3902145f0b30961b19680bdf4/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:12.234992 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84d3224-a163-4d83-b835-055e51b09b27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6b039a350343464abc724850d42ace5c, user_id=130a6da3902145f0b30961b19680bdf4, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:12.237957 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84d3224-a163-4d83-b835-055e51b09b27 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:12.253945 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1430/2827] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:50:12 2022] PUT /identity/v3/projects/6b039a350343464abc724850d42ace5c/users/130a6da3902145f0b30961b19680bdf4/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:12.267236 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d431a4-d238-45af-9536-5d3432f3c3f7 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:12.267737 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d431a4-d238-45af-9536-5d3432f3c3f7 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:12.267893 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d431a4-d238-45af-9536-5d3432f3c3f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:12.298136 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-26d431a4-d238-45af-9536-5d3432f3c3f7 None None] MFA Rules not processed for user `130a6da3902145f0b30961b19680bdf4`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:12.299352 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-26d431a4-d238-45af-9536-5d3432f3c3f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:12.365954 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1398/2828] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:12 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:12.371469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84d3224-a163-4d83-b835-055e51b09b27 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:12.440483 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84d3224-a163-4d83-b835-055e51b09b27 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:12.445786 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cc7346-961e-43db-b66e-525ef1d54891 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:12.446382 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cc7346-961e-43db-b66e-525ef1d54891 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:12.446661 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cc7346-961e-43db-b66e-525ef1d54891 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:12.446932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cc7346-961e-43db-b66e-525ef1d54891 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:12.447563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2cc7346-961e-43db-b66e-525ef1d54891 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:12.449510 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2cc7346-961e-43db-b66e-525ef1d54891 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:12.485427 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1431/2829] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:50:12 2022] POST /identity/v3/projects => generated 399 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:50:12.486614 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d431a4-d238-45af-9536-5d3432f3c3f7 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:12.498760 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d431a4-d238-45af-9536-5d3432f3c3f7 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:12.560651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8e5fae-0661-4f40-af77-4b78b3306acf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:12.560651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8e5fae-0661-4f40-af77-4b78b3306acf admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:12.560651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8e5fae-0661-4f40-af77-4b78b3306acf admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:12.560651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8e5fae-0661-4f40-af77-4b78b3306acf admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:12.561130 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8e5fae-0661-4f40-af77-4b78b3306acf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:12.562224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8e5fae-0661-4f40-af77-4b78b3306acf admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:12.571478 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:12.583248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1399/2830] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:50:12 2022] POST /identity/v3/users => generated 306 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:12.588782 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cc7346-961e-43db-b66e-525ef1d54891 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:12.599925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cc7346-961e-43db-b66e-525ef1d54891 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:12.606125 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15258a44-c0b0-422f-b752-69212d88678d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:12.606822 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15258a44-c0b0-422f-b752-69212d88678d admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:12.608655 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15258a44-c0b0-422f-b752-69212d88678d admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:12.609093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15258a44-c0b0-422f-b752-69212d88678d admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:12.609780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15258a44-c0b0-422f-b752-69212d88678d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:12.611768 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15258a44-c0b0-422f-b752-69212d88678d admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:12.623558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1432/2831] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:12 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:12.627812 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8e5fae-0661-4f40-af77-4b78b3306acf admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:12.638018 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8e5fae-0661-4f40-af77-4b78b3306acf admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:12.642780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18d0e7c-fad1-41ae-9201-83bc4df7fced admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:12.643421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18d0e7c-fad1-41ae-9201-83bc4df7fced admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:12.643701 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18d0e7c-fad1-41ae-9201-83bc4df7fced admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:12.643976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18d0e7c-fad1-41ae-9201-83bc4df7fced admin admin] PATH_INFO: `/v3/projects/2684aa4c9a424c9f9b632e1369c2cdec/users/7bee67b05b1d4846bf780d5410c2e95b/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:12.661235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18d0e7c-fad1-41ae-9201-83bc4df7fced admin admin] RBAC: Authorizing `identity:create_grant(project_id=2684aa4c9a424c9f9b632e1369c2cdec, user_id=7bee67b05b1d4846bf780d5410c2e95b, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:12.664628 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18d0e7c-fad1-41ae-9201-83bc4df7fced admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:12.678725 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1400/2832] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:50:12 2022] PUT /identity/v3/projects/2684aa4c9a424c9f9b632e1369c2cdec/users/7bee67b05b1d4846bf780d5410c2e95b/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:12.683338 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15258a44-c0b0-422f-b752-69212d88678d admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:12.693360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15258a44-c0b0-422f-b752-69212d88678d admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:12.752927 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd105136-4165-4667-aa17-565bdc802f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:12.753554 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd105136-4165-4667-aa17-565bdc802f6c admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:12.753835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd105136-4165-4667-aa17-565bdc802f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:12.754109 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd105136-4165-4667-aa17-565bdc802f6c admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:12.754800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd105136-4165-4667-aa17-565bdc802f6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:12.756438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd105136-4165-4667-aa17-565bdc802f6c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:12.766915 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1433/2833] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:12 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:12.770789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18d0e7c-fad1-41ae-9201-83bc4df7fced admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:12.781030 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18d0e7c-fad1-41ae-9201-83bc4df7fced admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:12.785594 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551a275e-105e-44fd-b590-9252db5d732c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:12.786254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551a275e-105e-44fd-b590-9252db5d732c admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:12.786528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551a275e-105e-44fd-b590-9252db5d732c admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:12.786797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551a275e-105e-44fd-b590-9252db5d732c admin admin] PATH_INFO: `/v3/projects/2684aa4c9a424c9f9b632e1369c2cdec/users/7bee67b05b1d4846bf780d5410c2e95b/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:12.789239 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551a275e-105e-44fd-b590-9252db5d732c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2684aa4c9a424c9f9b632e1369c2cdec, user_id=7bee67b05b1d4846bf780d5410c2e95b, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:12.791986 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551a275e-105e-44fd-b590-9252db5d732c admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:12.803677 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1401/2834] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:50:12 2022] PUT /identity/v3/projects/2684aa4c9a424c9f9b632e1369c2cdec/users/7bee67b05b1d4846bf780d5410c2e95b/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:12.814317 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a8ea77-fb8c-46a0-9b9a-4f70bc7bbbfd None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:12.814603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a8ea77-fb8c-46a0-9b9a-4f70bc7bbbfd None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:12.814878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a8ea77-fb8c-46a0-9b9a-4f70bc7bbbfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:12.834446 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:12.845880 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-13a8ea77-fb8c-46a0-9b9a-4f70bc7bbbfd None None] MFA Rules not processed for user `7bee67b05b1d4846bf780d5410c2e95b`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:12.847113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-13a8ea77-fb8c-46a0-9b9a-4f70bc7bbbfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:12.916771 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1434/2835] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:12 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:12.928165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551a275e-105e-44fd-b590-9252db5d732c admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:12.930042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-551a275e-105e-44fd-b590-9252db5d732c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:12.990718 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551a275e-105e-44fd-b590-9252db5d732c admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:12.995072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de8cb5f-b51a-41d3-899b-217896f37456 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:12.995669 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de8cb5f-b51a-41d3-899b-217896f37456 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:12.995941 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de8cb5f-b51a-41d3-899b-217896f37456 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:12.996204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de8cb5f-b51a-41d3-899b-217896f37456 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:13.005125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de8cb5f-b51a-41d3-899b-217896f37456 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:13.006553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de8cb5f-b51a-41d3-899b-217896f37456 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:13.015754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1402/2836] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:12 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:13.093603 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a8ea77-fb8c-46a0-9b9a-4f70bc7bbbfd None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:13.103246 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a8ea77-fb8c-46a0-9b9a-4f70bc7bbbfd None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:13.106828 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f63caf5-5e30-4925-8f1b-6e4ca0ceaeba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:13.107416 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f63caf5-5e30-4925-8f1b-6e4ca0ceaeba service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:13.107690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f63caf5-5e30-4925-8f1b-6e4ca0ceaeba service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:13.107958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f63caf5-5e30-4925-8f1b-6e4ca0ceaeba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:13.168180 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f63caf5-5e30-4925-8f1b-6e4ca0ceaeba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:13.169569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f63caf5-5e30-4925-8f1b-6e4ca0ceaeba service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:13.178837 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1435/2837] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:13 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:17.049509 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de8cb5f-b51a-41d3-899b-217896f37456 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:17.050993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1de8cb5f-b51a-41d3-899b-217896f37456 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:17.139135 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de8cb5f-b51a-41d3-899b-217896f37456 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:17.145534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5103a70e-ebd2-453e-8acb-2e46ffc91087 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:17.146220 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5103a70e-ebd2-453e-8acb-2e46ffc91087 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:17.146662 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5103a70e-ebd2-453e-8acb-2e46ffc91087 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:17.146942 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5103a70e-ebd2-453e-8acb-2e46ffc91087 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:17.148800 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-5103a70e-ebd2-453e-8acb-2e46ffc91087 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:17.163653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5103a70e-ebd2-453e-8acb-2e46ffc91087 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:17.165885 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5103a70e-ebd2-453e-8acb-2e46ffc91087 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:17.177674 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1403/2838] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:17.280093 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f63caf5-5e30-4925-8f1b-6e4ca0ceaeba service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:17.351263 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f63caf5-5e30-4925-8f1b-6e4ca0ceaeba service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:17.356260 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477ac4ad-6231-41dd-a710-3ac6b8c419fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:17.356911 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477ac4ad-6231-41dd-a710-3ac6b8c419fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:17.357191 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477ac4ad-6231-41dd-a710-3ac6b8c419fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:17.358180 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477ac4ad-6231-41dd-a710-3ac6b8c419fa admin admin] PATH_INFO: `/v3/users/130a6da3902145f0b30961b19680bdf4` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:17.359355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477ac4ad-6231-41dd-a710-3ac6b8c419fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=130a6da3902145f0b30961b19680bdf4)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:17.361933 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477ac4ad-6231-41dd-a710-3ac6b8c419fa admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:17.402274 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:17.412606 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-477ac4ad-6231-41dd-a710-3ac6b8c419fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '130a6da3902145f0b30961b19680bdf4'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:17.413799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-477ac4ad-6231-41dd-a710-3ac6b8c419fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '130a6da3902145f0b30961b19680bdf4'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:17.421571 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-477ac4ad-6231-41dd-a710-3ac6b8c419fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '130a6da3902145f0b30961b19680bdf4'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:17.422749 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-477ac4ad-6231-41dd-a710-3ac6b8c419fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '130a6da3902145f0b30961b19680bdf4'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:17.435882 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-477ac4ad-6231-41dd-a710-3ac6b8c419fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '130a6da3902145f0b30961b19680bdf4'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:17.449504 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1436/2839] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:50:17 2022] DELETE /identity/v3/users/130a6da3902145f0b30961b19680bdf4 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:17.453867 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951cf262-e510-44c4-ad2d-fe6c79676787 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:17.454153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951cf262-e510-44c4-ad2d-fe6c79676787 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:17.454431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951cf262-e510-44c4-ad2d-fe6c79676787 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:17.473916 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-951cf262-e510-44c4-ad2d-fe6c79676787 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:17.475179 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-951cf262-e510-44c4-ad2d-fe6c79676787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:17.525204 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:17.550113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1404/2840] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:17.560553 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477ac4ad-6231-41dd-a710-3ac6b8c419fa admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:17.561960 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-477ac4ad-6231-41dd-a710-3ac6b8c419fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:17.619341 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477ac4ad-6231-41dd-a710-3ac6b8c419fa admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:17.622996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3eb2579-d367-4e1d-aa66-011aaab6dede service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:17.623591 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3eb2579-d367-4e1d-aa66-011aaab6dede service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:17.623875 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3eb2579-d367-4e1d-aa66-011aaab6dede service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:17.624147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3eb2579-d367-4e1d-aa66-011aaab6dede service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:17.633726 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3eb2579-d367-4e1d-aa66-011aaab6dede service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:17.635139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3eb2579-d367-4e1d-aa66-011aaab6dede service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:17.644476 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1437/2841] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:17.912078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951cf262-e510-44c4-ad2d-fe6c79676787 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:17.914167 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-951cf262-e510-44c4-ad2d-fe6c79676787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:17.938808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951cf262-e510-44c4-ad2d-fe6c79676787 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:17.938808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:17.938808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:17.938808 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:17.939186 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] PATH_INFO: `/v3/projects/6b039a350343464abc724850d42ace5c` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:17.939501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] RBAC: Authorizing `identity:delete_project(project_id=6b039a350343464abc724850d42ace5c)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:17.942957 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:17.999657 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6b039a350343464abc724850d42ace5c', 'request_id': 'req-40cc0fd8-45fb-4034-8a05-7821b77832af'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.008212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6b039a350343464abc724850d42ace5c', 'request_id': 'req-40cc0fd8-45fb-4034-8a05-7821b77832af'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.017546 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6b039a350343464abc724850d42ace5c', 'request_id': 'req-40cc0fd8-45fb-4034-8a05-7821b77832af'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.029456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] The token cache is being invalidate because project 6b039a350343464abc724850d42ace5c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:50:18.029748 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.030398 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.031887 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1405/2842] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:50:17 2022] DELETE /identity/v3/projects/6b039a350343464abc724850d42ace5c => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:18.035991 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3eb2579-d367-4e1d-aa66-011aaab6dede service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:18.046823 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3eb2579-d367-4e1d-aa66-011aaab6dede service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:18.105334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ae08f1-5084-46b3-a1d1-e23eca5021cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:18.106043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ae08f1-5084-46b3-a1d1-e23eca5021cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:18.106386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ae08f1-5084-46b3-a1d1-e23eca5021cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:18.106723 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ae08f1-5084-46b3-a1d1-e23eca5021cb admin admin] PATH_INFO: `/v3/users/7bee67b05b1d4846bf780d5410c2e95b` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:18.107904 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ae08f1-5084-46b3-a1d1-e23eca5021cb admin admin] RBAC: Authorizing `identity:delete_user(user_id=7bee67b05b1d4846bf780d5410c2e95b)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:18.109889 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ae08f1-5084-46b3-a1d1-e23eca5021cb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:18.155866 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-54ae08f1-5084-46b3-a1d1-e23eca5021cb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7bee67b05b1d4846bf780d5410c2e95b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.156532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-54ae08f1-5084-46b3-a1d1-e23eca5021cb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7bee67b05b1d4846bf780d5410c2e95b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.163951 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-54ae08f1-5084-46b3-a1d1-e23eca5021cb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7bee67b05b1d4846bf780d5410c2e95b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.164854 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-54ae08f1-5084-46b3-a1d1-e23eca5021cb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7bee67b05b1d4846bf780d5410c2e95b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.165790 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:18.180587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-54ae08f1-5084-46b3-a1d1-e23eca5021cb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7bee67b05b1d4846bf780d5410c2e95b'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.196187 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1438/2843] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:50:18 2022] DELETE /identity/v3/users/7bee67b05b1d4846bf780d5410c2e95b => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:50:18.462057 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:18.463531 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:18.477357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cc0fd8-45fb-4034-8a05-7821b77832af admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:18.543774 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b486c06-c1b7-4410-b9b4-28063096d52d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:18.544421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b486c06-c1b7-4410-b9b4-28063096d52d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:18.544701 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b486c06-c1b7-4410-b9b4-28063096d52d admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:18.544973 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b486c06-c1b7-4410-b9b4-28063096d52d admin admin] PATH_INFO: `/v3/projects/2684aa4c9a424c9f9b632e1369c2cdec` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:18.546074 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b486c06-c1b7-4410-b9b4-28063096d52d admin admin] RBAC: Authorizing `identity:delete_project(project_id=2684aa4c9a424c9f9b632e1369c2cdec)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:18.548073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b486c06-c1b7-4410-b9b4-28063096d52d admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:18.592424 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7b486c06-c1b7-4410-b9b4-28063096d52d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2684aa4c9a424c9f9b632e1369c2cdec', 'request_id': 'req-7b486c06-c1b7-4410-b9b4-28063096d52d'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.597930 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7b486c06-c1b7-4410-b9b4-28063096d52d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2684aa4c9a424c9f9b632e1369c2cdec', 'request_id': 'req-7b486c06-c1b7-4410-b9b4-28063096d52d'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.606241 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7b486c06-c1b7-4410-b9b4-28063096d52d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2684aa4c9a424c9f9b632e1369c2cdec', 'request_id': 'req-7b486c06-c1b7-4410-b9b4-28063096d52d'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.608201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:18.619334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7b486c06-c1b7-4410-b9b4-28063096d52d admin admin] The token cache is being invalidate because project 2684aa4c9a424c9f9b632e1369c2cdec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:50:18.619770 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7b486c06-c1b7-4410-b9b4-28063096d52d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.620457 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-7b486c06-c1b7-4410-b9b4-28063096d52d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:18.621965 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1406/2844] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:50:18 2022] DELETE /identity/v3/projects/2684aa4c9a424c9f9b632e1369c2cdec => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:18.627197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef46627f-a7a4-4186-be1a-ae28b2e2ff8f None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:18.627670 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef46627f-a7a4-4186-be1a-ae28b2e2ff8f None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:18.627783 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef46627f-a7a4-4186-be1a-ae28b2e2ff8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:18.648701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-ef46627f-a7a4-4186-be1a-ae28b2e2ff8f None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:18.649900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ef46627f-a7a4-4186-be1a-ae28b2e2ff8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:18.706922 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1439/2845] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:18.716765 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06950aa1-bea2-4283-b0ca-79ea2997fd65 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:18.717053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06950aa1-bea2-4283-b0ca-79ea2997fd65 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:18.717335 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06950aa1-bea2-4283-b0ca-79ea2997fd65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:18.738185 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-06950aa1-bea2-4283-b0ca-79ea2997fd65 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:18.739396 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-06950aa1-bea2-4283-b0ca-79ea2997fd65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:18.748095 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1407/2846] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:18.752508 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef46627f-a7a4-4186-be1a-ae28b2e2ff8f None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:18.767343 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef46627f-a7a4-4186-be1a-ae28b2e2ff8f None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:18.772144 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef83b0c-c0c3-45e3-b036-a001cd91b7ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:18.773417 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef83b0c-c0c3-45e3-b036-a001cd91b7ae admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:18.773694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef83b0c-c0c3-45e3-b036-a001cd91b7ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:18.773964 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef83b0c-c0c3-45e3-b036-a001cd91b7ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:18.774618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef83b0c-c0c3-45e3-b036-a001cd91b7ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81593) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:50:18.774923 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef83b0c-c0c3-45e3-b036-a001cd91b7ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:18.776546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef83b0c-c0c3-45e3-b036-a001cd91b7ae admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:18.788559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1440/2847] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:50:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:50:18.792493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06950aa1-bea2-4283-b0ca-79ea2997fd65 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:18.801582 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06950aa1-bea2-4283-b0ca-79ea2997fd65 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:18.806452 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73db16c4-5a1e-400c-b10d-05b5229ddbff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:18.807034 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73db16c4-5a1e-400c-b10d-05b5229ddbff admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:18.807310 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73db16c4-5a1e-400c-b10d-05b5229ddbff admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:18.807576 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73db16c4-5a1e-400c-b10d-05b5229ddbff admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:18.808193 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73db16c4-5a1e-400c-b10d-05b5229ddbff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:18.810109 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73db16c4-5a1e-400c-b10d-05b5229ddbff admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:18.837825 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1408/2848] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:50:18 2022] POST /identity/v3/projects => generated 377 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:18.842238 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef83b0c-c0c3-45e3-b036-a001cd91b7ae admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:18.852855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef83b0c-c0c3-45e3-b036-a001cd91b7ae admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:18.909073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49863ff-b16d-45f6-b2d1-d9de4e86179e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:18.909676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49863ff-b16d-45f6-b2d1-d9de4e86179e admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:18.909955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49863ff-b16d-45f6-b2d1-d9de4e86179e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:18.910229 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49863ff-b16d-45f6-b2d1-d9de4e86179e admin admin] PATH_INFO: `/v3/users` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:18.910851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49863ff-b16d-45f6-b2d1-d9de4e86179e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:18.912800 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49863ff-b16d-45f6-b2d1-d9de4e86179e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:18.934041 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1441/2849] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:50:18 2022] POST /identity/v3/users => generated 289 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:18.938298 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73db16c4-5a1e-400c-b10d-05b5229ddbff admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:18.950489 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73db16c4-5a1e-400c-b10d-05b5229ddbff admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:18.955612 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaf5cf9-b212-452f-8778-6e75b74b856d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:18.955861 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaf5cf9-b212-452f-8778-6e75b74b856d admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:18.956084 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaf5cf9-b212-452f-8778-6e75b74b856d admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:18.956389 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaf5cf9-b212-452f-8778-6e75b74b856d admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:18.957163 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eaf5cf9-b212-452f-8778-6e75b74b856d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:18.958786 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eaf5cf9-b212-452f-8778-6e75b74b856d admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:18.970273 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1409/2850] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:18 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:18.974487 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49863ff-b16d-45f6-b2d1-d9de4e86179e admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:18.985429 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49863ff-b16d-45f6-b2d1-d9de4e86179e admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:18.990491 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e78b35-7cad-4735-b0bc-78335475e513 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:18.991087 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e78b35-7cad-4735-b0bc-78335475e513 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:18.991968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e78b35-7cad-4735-b0bc-78335475e513 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:18.992246 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e78b35-7cad-4735-b0bc-78335475e513 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:18.992925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e78b35-7cad-4735-b0bc-78335475e513 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:18.994505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e78b35-7cad-4735-b0bc-78335475e513 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:19.005436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1442/2851] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:19.009454 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaf5cf9-b212-452f-8778-6e75b74b856d admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:19.019205 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaf5cf9-b212-452f-8778-6e75b74b856d admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:19.023827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9a54e4-9b30-4424-bddf-f3b24e32187f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:19.024514 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9a54e4-9b30-4424-bddf-f3b24e32187f admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:19.024793 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9a54e4-9b30-4424-bddf-f3b24e32187f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:19.025068 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9a54e4-9b30-4424-bddf-f3b24e32187f admin admin] PATH_INFO: `/v3/projects/4529386413fe443694d87b89621835f2/users/ec2a94b102e44fc38071709f2ea2ab75/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:19.039622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9a54e4-9b30-4424-bddf-f3b24e32187f admin admin] RBAC: Authorizing `identity:create_grant(project_id=4529386413fe443694d87b89621835f2, user_id=ec2a94b102e44fc38071709f2ea2ab75, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:19.042306 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9a54e4-9b30-4424-bddf-f3b24e32187f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:19.055349 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1410/2852] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:50:19 2022] PUT /identity/v3/projects/4529386413fe443694d87b89621835f2/users/ec2a94b102e44fc38071709f2ea2ab75/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:19.065631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae671748-3338-4fc4-b43d-783d513e6bdc None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:19.065919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae671748-3338-4fc4-b43d-783d513e6bdc None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:19.066243 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae671748-3338-4fc4-b43d-783d513e6bdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:19.094729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-ae671748-3338-4fc4-b43d-783d513e6bdc None None] MFA Rules not processed for user `ec2a94b102e44fc38071709f2ea2ab75`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:19.095934 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ae671748-3338-4fc4-b43d-783d513e6bdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:19.126141 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:19.165806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1443/2853] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:19 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:19.177137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9a54e4-9b30-4424-bddf-f3b24e32187f admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:19.178894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-9e9a54e4-9b30-4424-bddf-f3b24e32187f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:19.251700 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9a54e4-9b30-4424-bddf-f3b24e32187f admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:19.255753 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7173988e-96aa-43bd-99a1-174a5ea6e501 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:19.256392 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7173988e-96aa-43bd-99a1-174a5ea6e501 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:19.256723 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7173988e-96aa-43bd-99a1-174a5ea6e501 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:19.257025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7173988e-96aa-43bd-99a1-174a5ea6e501 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:19.266587 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7173988e-96aa-43bd-99a1-174a5ea6e501 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:19.268025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7173988e-96aa-43bd-99a1-174a5ea6e501 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:19.280070 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1411/2854] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:19 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:20.672884 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae671748-3338-4fc4-b43d-783d513e6bdc None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:20.674356 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ae671748-3338-4fc4-b43d-783d513e6bdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:20.736534 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae671748-3338-4fc4-b43d-783d513e6bdc None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:20.740621 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0064d8a-fc86-44e3-89f3-d126e5aceacd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:20.742135 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0064d8a-fc86-44e3-89f3-d126e5aceacd service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:20.742717 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0064d8a-fc86-44e3-89f3-d126e5aceacd service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:20.743072 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0064d8a-fc86-44e3-89f3-d126e5aceacd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:20.745248 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-b0064d8a-fc86-44e3-89f3-d126e5aceacd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:20.822360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0064d8a-fc86-44e3-89f3-d126e5aceacd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:20.824156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0064d8a-fc86-44e3-89f3-d126e5aceacd service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:20.835649 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1444/2855] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:20 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:29.274005 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=55,keystone:INSERT=2 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:30.831043 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=64 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:36.994338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b612439a-0c2c-4197-93f5-999733ad2ee6 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:36.994895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b612439a-0c2c-4197-93f5-999733ad2ee6 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:36.994895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b612439a-0c2c-4197-93f5-999733ad2ee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:37.014295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-b612439a-0c2c-4197-93f5-999733ad2ee6 None None] MFA Rules not processed for user `4d70d61439324d4b9c3b3513799b896c`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:37.015483 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-b612439a-0c2c-4197-93f5-999733ad2ee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:37.071437 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1412/2856] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:36 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:37.399402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0064d8a-fc86-44e3-89f3-d126e5aceacd service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:37.407910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0064d8a-fc86-44e3-89f3-d126e5aceacd service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:37.411631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7efa058-5694-4d9c-88bc-59e4e64a0662 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:37.412232 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7efa058-5694-4d9c-88bc-59e4e64a0662 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:37.412536 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7efa058-5694-4d9c-88bc-59e4e64a0662 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:37.412809 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7efa058-5694-4d9c-88bc-59e4e64a0662 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:37.415136 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c7efa058-5694-4d9c-88bc-59e4e64a0662 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:37.425048 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7efa058-5694-4d9c-88bc-59e4e64a0662 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:37.426453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7efa058-5694-4d9c-88bc-59e4e64a0662 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:37.434449 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1445/2857] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:37 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:46.244738 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b612439a-0c2c-4197-93f5-999733ad2ee6 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:46.304653 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b612439a-0c2c-4197-93f5-999733ad2ee6 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:46.309130 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdc6e4e-b9cc-44a7-a111-5695b5801836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:46.309741 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdc6e4e-b9cc-44a7-a111-5695b5801836 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:46.310015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdc6e4e-b9cc-44a7-a111-5695b5801836 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:46.310281 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdc6e4e-b9cc-44a7-a111-5695b5801836 admin admin] PATH_INFO: `/v3/users/ec2a94b102e44fc38071709f2ea2ab75` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:46.311374 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cdc6e4e-b9cc-44a7-a111-5695b5801836 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec2a94b102e44fc38071709f2ea2ab75)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:46.313307 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cdc6e4e-b9cc-44a7-a111-5695b5801836 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:46.358322 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1cdc6e4e-b9cc-44a7-a111-5695b5801836 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec2a94b102e44fc38071709f2ea2ab75'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:46.358957 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1cdc6e4e-b9cc-44a7-a111-5695b5801836 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec2a94b102e44fc38071709f2ea2ab75'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:46.359570 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1cdc6e4e-b9cc-44a7-a111-5695b5801836 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec2a94b102e44fc38071709f2ea2ab75'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:46.366704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1cdc6e4e-b9cc-44a7-a111-5695b5801836 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec2a94b102e44fc38071709f2ea2ab75'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:46.379157 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-1cdc6e4e-b9cc-44a7-a111-5695b5801836 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec2a94b102e44fc38071709f2ea2ab75'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:46.391101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1413/2858] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:50:46 2022] DELETE /identity/v3/users/ec2a94b102e44fc38071709f2ea2ab75 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:46.395703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5993ee2-feba-46b0-a249-3bf63a40554c None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:46.396155 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5993ee2-feba-46b0-a249-3bf63a40554c None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:46.396268 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5993ee2-feba-46b0-a249-3bf63a40554c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:46.415230 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-c5993ee2-feba-46b0-a249-3bf63a40554c None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:46.416437 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c5993ee2-feba-46b0-a249-3bf63a40554c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:46.473313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1446/2859] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:46.483888 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdc6e4e-b9cc-44a7-a111-5695b5801836 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:46.485418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-1cdc6e4e-b9cc-44a7-a111-5695b5801836 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:46.513637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:46.544827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdc6e4e-b9cc-44a7-a111-5695b5801836 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:46.548553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52409302-c802-4ce9-92f0-5cca7330feb4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:46.549153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52409302-c802-4ce9-92f0-5cca7330feb4 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:46.549430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52409302-c802-4ce9-92f0-5cca7330feb4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:46.549701 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52409302-c802-4ce9-92f0-5cca7330feb4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:46.558536 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52409302-c802-4ce9-92f0-5cca7330feb4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:46.559959 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52409302-c802-4ce9-92f0-5cca7330feb4 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:46.569163 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1414/2860] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:50:46.830865 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5993ee2-feba-46b0-a249-3bf63a40554c None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:46.832465 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-c5993ee2-feba-46b0-a249-3bf63a40554c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:46.847210 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5993ee2-feba-46b0-a249-3bf63a40554c None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:46.852246 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:46.852982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:46.853262 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:46.853535 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e admin admin] PATH_INFO: `/v3/projects/4529386413fe443694d87b89621835f2` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:46.854671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e admin admin] RBAC: Authorizing `identity:delete_project(project_id=4529386413fe443694d87b89621835f2)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:46.856581 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:46.913110 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4529386413fe443694d87b89621835f2', 'request_id': 'req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:46.918659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4529386413fe443694d87b89621835f2', 'request_id': 'req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:46.926243 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4529386413fe443694d87b89621835f2', 'request_id': 'req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:46.938359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e admin admin] The token cache is being invalidate because project 4529386413fe443694d87b89621835f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:50:46.938647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:46.939399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c1151aa6-d223-4c99-a8ea-cc8e6cb6e93e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:46.941156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1447/2861] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:50:46 2022] DELETE /identity/v3/projects/4529386413fe443694d87b89621835f2 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:46.946382 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ca4385-a163-4978-994f-346403763320 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:46.946681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ca4385-a163-4978-994f-346403763320 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:46.947015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ca4385-a163-4978-994f-346403763320 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:46.965981 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-43ca4385-a163-4978-994f-346403763320 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:46.967290 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-43ca4385-a163-4978-994f-346403763320 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:47.029357 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1415/2862] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:47.037279 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e287d6d0-c3d5-4410-bf8c-073bde047476 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.037566 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e287d6d0-c3d5-4410-bf8c-073bde047476 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.037845 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e287d6d0-c3d5-4410-bf8c-073bde047476 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.057574 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-e287d6d0-c3d5-4410-bf8c-073bde047476 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:47.058774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e287d6d0-c3d5-4410-bf8c-073bde047476 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:47.066561 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1448/2863] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:47.071292 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ca4385-a163-4978-994f-346403763320 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:47.080870 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ca4385-a163-4978-994f-346403763320 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:47.085378 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5a2a5c-9999-4007-b94d-3e53b137d7b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:47.085960 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5a2a5c-9999-4007-b94d-3e53b137d7b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.086240 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5a2a5c-9999-4007-b94d-3e53b137d7b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.086506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5a2a5c-9999-4007-b94d-3e53b137d7b1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.087138 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5a2a5c-9999-4007-b94d-3e53b137d7b1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:50:47.087433 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5a2a5c-9999-4007-b94d-3e53b137d7b1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:47.089047 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5a2a5c-9999-4007-b94d-3e53b137d7b1 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:47.100113 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1416/2864] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:50:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:50:47.105650 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e287d6d0-c3d5-4410-bf8c-073bde047476 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:47.115065 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e287d6d0-c3d5-4410-bf8c-073bde047476 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:47.120169 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a3dbb1-a4c2-4da5-b2c3-6e440f2d1627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:47.120795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a3dbb1-a4c2-4da5-b2c3-6e440f2d1627 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.121076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a3dbb1-a4c2-4da5-b2c3-6e440f2d1627 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.121349 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a3dbb1-a4c2-4da5-b2c3-6e440f2d1627 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.121973 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a3dbb1-a4c2-4da5-b2c3-6e440f2d1627 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:47.123846 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a3dbb1-a4c2-4da5-b2c3-6e440f2d1627 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:47.150604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1449/2865] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:50:47 2022] POST /identity/v3/projects => generated 375 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:47.154663 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5a2a5c-9999-4007-b94d-3e53b137d7b1 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:47.164549 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5a2a5c-9999-4007-b94d-3e53b137d7b1 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:47.220033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bcb724-8814-4384-bb83-91178756b47a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:47.220650 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bcb724-8814-4384-bb83-91178756b47a admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.220929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bcb724-8814-4384-bb83-91178756b47a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.221201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bcb724-8814-4384-bb83-91178756b47a admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.221823 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bcb724-8814-4384-bb83-91178756b47a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:47.223731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bcb724-8814-4384-bb83-91178756b47a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:47.245516 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1417/2866] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:50:47 2022] POST /identity/v3/users => generated 288 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:47.249590 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a3dbb1-a4c2-4da5-b2c3-6e440f2d1627 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:47.259542 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a3dbb1-a4c2-4da5-b2c3-6e440f2d1627 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:47.264067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e770cf-11b1-4d1e-b093-d21daaf6e7f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:47.264690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e770cf-11b1-4d1e-b093-d21daaf6e7f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.264969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e770cf-11b1-4d1e-b093-d21daaf6e7f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.265243 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e770cf-11b1-4d1e-b093-d21daaf6e7f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.265883 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e770cf-11b1-4d1e-b093-d21daaf6e7f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:47.267451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e770cf-11b1-4d1e-b093-d21daaf6e7f2 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:47.277847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1450/2867] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:47 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:47.281572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bcb724-8814-4384-bb83-91178756b47a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:47.291725 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bcb724-8814-4384-bb83-91178756b47a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:47.296557 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae1cf87-b7fa-4ae5-8ac1-be6cc92cc191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:47.297145 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae1cf87-b7fa-4ae5-8ac1-be6cc92cc191 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.297416 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae1cf87-b7fa-4ae5-8ac1-be6cc92cc191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.297682 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae1cf87-b7fa-4ae5-8ac1-be6cc92cc191 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.298313 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae1cf87-b7fa-4ae5-8ac1-be6cc92cc191 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:47.299928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae1cf87-b7fa-4ae5-8ac1-be6cc92cc191 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:47.310263 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1418/2868] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:47 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:47.314139 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e770cf-11b1-4d1e-b093-d21daaf6e7f2 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:47.324334 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e770cf-11b1-4d1e-b093-d21daaf6e7f2 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:47.328867 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129547c5-898c-4713-b6d1-fc7daa864f46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:47.329497 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129547c5-898c-4713-b6d1-fc7daa864f46 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.329788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129547c5-898c-4713-b6d1-fc7daa864f46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.330044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129547c5-898c-4713-b6d1-fc7daa864f46 admin admin] PATH_INFO: `/v3/projects/3df28f392e2842b496fe6ea6c0614103/users/b8a9db6d79774f75b6fb8a4843c4ee27/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.344616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129547c5-898c-4713-b6d1-fc7daa864f46 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3df28f392e2842b496fe6ea6c0614103, user_id=b8a9db6d79774f75b6fb8a4843c4ee27, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:47.347219 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129547c5-898c-4713-b6d1-fc7daa864f46 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:47.359959 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1451/2869] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:50:47 2022] PUT /identity/v3/projects/3df28f392e2842b496fe6ea6c0614103/users/b8a9db6d79774f75b6fb8a4843c4ee27/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:47.371083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabe7791-36ac-43ec-8fe4-94efa60c2d97 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.372089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabe7791-36ac-43ec-8fe4-94efa60c2d97 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.372442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabe7791-36ac-43ec-8fe4-94efa60c2d97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.401234 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-dabe7791-36ac-43ec-8fe4-94efa60c2d97 None None] MFA Rules not processed for user `b8a9db6d79774f75b6fb8a4843c4ee27`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:47.402448 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-dabe7791-36ac-43ec-8fe4-94efa60c2d97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:47.423157 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:47.466571 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1419/2870] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:47 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:47.471134 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129547c5-898c-4713-b6d1-fc7daa864f46 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:47.480594 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:47.536003 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129547c5-898c-4713-b6d1-fc7daa864f46 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:47.541291 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebbdfa4-3def-4985-9225-de27a94d75d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:47.541887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebbdfa4-3def-4985-9225-de27a94d75d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.542168 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebbdfa4-3def-4985-9225-de27a94d75d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.542443 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebbdfa4-3def-4985-9225-de27a94d75d3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.543110 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebbdfa4-3def-4985-9225-de27a94d75d3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:47.545037 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebbdfa4-3def-4985-9225-de27a94d75d3 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:47.571063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1452/2871] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:50:47 2022] POST /identity/v3/projects => generated 375 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:47.575114 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabe7791-36ac-43ec-8fe4-94efa60c2d97 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:47.586430 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabe7791-36ac-43ec-8fe4-94efa60c2d97 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:47.642190 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca54702-e3a9-49b5-a1f5-f8039419d738 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:47.642790 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca54702-e3a9-49b5-a1f5-f8039419d738 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.643072 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca54702-e3a9-49b5-a1f5-f8039419d738 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.643347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca54702-e3a9-49b5-a1f5-f8039419d738 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.643985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca54702-e3a9-49b5-a1f5-f8039419d738 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:47.645956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca54702-e3a9-49b5-a1f5-f8039419d738 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:47.667906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1420/2872] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:50:47 2022] POST /identity/v3/users => generated 288 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:47.671887 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebbdfa4-3def-4985-9225-de27a94d75d3 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:47.682098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebbdfa4-3def-4985-9225-de27a94d75d3 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:47.686906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430b7417-f6d0-40be-9297-b04f57a0fe97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:47.687499 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b7417-f6d0-40be-9297-b04f57a0fe97 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.687799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b7417-f6d0-40be-9297-b04f57a0fe97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.688069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b7417-f6d0-40be-9297-b04f57a0fe97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.688772 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430b7417-f6d0-40be-9297-b04f57a0fe97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:47.690330 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430b7417-f6d0-40be-9297-b04f57a0fe97 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:47.700656 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1453/2873] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:47 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:47.704389 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca54702-e3a9-49b5-a1f5-f8039419d738 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:47.714906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca54702-e3a9-49b5-a1f5-f8039419d738 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:47.719523 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a4d3eb-ee7b-4c12-8d43-ad2d0833eb48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:47.720102 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a4d3eb-ee7b-4c12-8d43-ad2d0833eb48 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.720419 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a4d3eb-ee7b-4c12-8d43-ad2d0833eb48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.720692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a4d3eb-ee7b-4c12-8d43-ad2d0833eb48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.721350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a4d3eb-ee7b-4c12-8d43-ad2d0833eb48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:47.722935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a4d3eb-ee7b-4c12-8d43-ad2d0833eb48 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:47.733210 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1421/2874] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:47 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:47.737133 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430b7417-f6d0-40be-9297-b04f57a0fe97 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:47.747170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430b7417-f6d0-40be-9297-b04f57a0fe97 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:47.751727 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eff939a-b8eb-40a0-bea2-41361345a0af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:47.752381 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eff939a-b8eb-40a0-bea2-41361345a0af admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.752660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eff939a-b8eb-40a0-bea2-41361345a0af admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.752932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eff939a-b8eb-40a0-bea2-41361345a0af admin admin] PATH_INFO: `/v3/projects/4abd488c1c394efbab6b06224b067fc0/users/22ad3dcc8adb4cfabe2f05a873454798/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.766802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eff939a-b8eb-40a0-bea2-41361345a0af admin admin] RBAC: Authorizing `identity:create_grant(project_id=4abd488c1c394efbab6b06224b067fc0, user_id=22ad3dcc8adb4cfabe2f05a873454798, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:47.769523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eff939a-b8eb-40a0-bea2-41361345a0af admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:47.782462 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1454/2875] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:50:47 2022] PUT /identity/v3/projects/4abd488c1c394efbab6b06224b067fc0/users/22ad3dcc8adb4cfabe2f05a873454798/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:47.793248 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43859f00-37ef-4758-9c9d-1b2445bebda6 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.793555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43859f00-37ef-4758-9c9d-1b2445bebda6 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.793849 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43859f00-37ef-4758-9c9d-1b2445bebda6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.822031 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-43859f00-37ef-4758-9c9d-1b2445bebda6 None None] MFA Rules not processed for user `22ad3dcc8adb4cfabe2f05a873454798`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:47.823230 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-43859f00-37ef-4758-9c9d-1b2445bebda6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:47.885390 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1422/2876] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:47 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:47.890259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eff939a-b8eb-40a0-bea2-41361345a0af admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:47.952721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eff939a-b8eb-40a0-bea2-41361345a0af admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:47.957440 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371f2dea-3d81-4f54-8143-0ed9927e71a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:47.958033 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371f2dea-3d81-4f54-8143-0ed9927e71a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:47.958312 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371f2dea-3d81-4f54-8143-0ed9927e71a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:47.958587 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371f2dea-3d81-4f54-8143-0ed9927e71a4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:47.959228 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371f2dea-3d81-4f54-8143-0ed9927e71a4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:47.961855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371f2dea-3d81-4f54-8143-0ed9927e71a4 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:47.988819 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1455/2877] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:50:47 2022] POST /identity/v3/projects => generated 377 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:47.993563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43859f00-37ef-4758-9c9d-1b2445bebda6 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:48.004137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43859f00-37ef-4758-9c9d-1b2445bebda6 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:48.033897 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:48.064970 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b210ac97-1938-44ac-98a0-92e6e089cc71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:48.065607 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b210ac97-1938-44ac-98a0-92e6e089cc71 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:48.065886 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b210ac97-1938-44ac-98a0-92e6e089cc71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:48.066155 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b210ac97-1938-44ac-98a0-92e6e089cc71 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:48.066789 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b210ac97-1938-44ac-98a0-92e6e089cc71 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:48.068740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b210ac97-1938-44ac-98a0-92e6e089cc71 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:48.090713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1423/2878] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:50:47 2022] POST /identity/v3/users => generated 295 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:48.095023 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371f2dea-3d81-4f54-8143-0ed9927e71a4 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:48.106029 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371f2dea-3d81-4f54-8143-0ed9927e71a4 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:48.110598 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd78168-d9ad-4a3b-8b94-fd8aeabef0d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:48.111193 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd78168-d9ad-4a3b-8b94-fd8aeabef0d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:48.111470 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd78168-d9ad-4a3b-8b94-fd8aeabef0d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:48.111743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd78168-d9ad-4a3b-8b94-fd8aeabef0d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:48.112406 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd78168-d9ad-4a3b-8b94-fd8aeabef0d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:48.114234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd78168-d9ad-4a3b-8b94-fd8aeabef0d7 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:48.126906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1456/2879] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:48 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:50:48.131508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b210ac97-1938-44ac-98a0-92e6e089cc71 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:48.141600 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b210ac97-1938-44ac-98a0-92e6e089cc71 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:48.146262 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca23a1d-74e1-401d-a95d-5c666a059854 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:48.146894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca23a1d-74e1-401d-a95d-5c666a059854 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:48.147171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca23a1d-74e1-401d-a95d-5c666a059854 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:48.147574 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca23a1d-74e1-401d-a95d-5c666a059854 admin admin] PATH_INFO: `/v3/projects/5f3b2a9c462542bbbd13c57d4e4e83c9/users/40284a885fd341af8e4a5274975d519e/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:48.162358 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca23a1d-74e1-401d-a95d-5c666a059854 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f3b2a9c462542bbbd13c57d4e4e83c9, user_id=40284a885fd341af8e4a5274975d519e, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:48.165641 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca23a1d-74e1-401d-a95d-5c666a059854 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:48.180467 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1424/2880] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:50:48 2022] PUT /identity/v3/projects/5f3b2a9c462542bbbd13c57d4e4e83c9/users/40284a885fd341af8e4a5274975d519e/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:48.184369 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd78168-d9ad-4a3b-8b94-fd8aeabef0d7 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:48.194505 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd78168-d9ad-4a3b-8b94-fd8aeabef0d7 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:48.222226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:48.251313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d149d0f-90cd-4c6f-8801-6b4e4eafebe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:48.251903 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d149d0f-90cd-4c6f-8801-6b4e4eafebe9 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:48.252186 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d149d0f-90cd-4c6f-8801-6b4e4eafebe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:48.252482 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d149d0f-90cd-4c6f-8801-6b4e4eafebe9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:48.253115 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d149d0f-90cd-4c6f-8801-6b4e4eafebe9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:48.254693 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d149d0f-90cd-4c6f-8801-6b4e4eafebe9 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:48.264080 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1457/2881] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:48 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:48.268442 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca23a1d-74e1-401d-a95d-5c666a059854 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:48.278806 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca23a1d-74e1-401d-a95d-5c666a059854 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:48.283589 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82a0332-ab31-4925-8f1b-0dfab5541692 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:48.284228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82a0332-ab31-4925-8f1b-0dfab5541692 admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:48.284539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82a0332-ab31-4925-8f1b-0dfab5541692 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:48.284814 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82a0332-ab31-4925-8f1b-0dfab5541692 admin admin] PATH_INFO: `/v3/projects/5f3b2a9c462542bbbd13c57d4e4e83c9/users/40284a885fd341af8e4a5274975d519e/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:48.286747 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82a0332-ab31-4925-8f1b-0dfab5541692 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f3b2a9c462542bbbd13c57d4e4e83c9, user_id=40284a885fd341af8e4a5274975d519e, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:48.290114 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82a0332-ab31-4925-8f1b-0dfab5541692 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:48.300154 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1425/2882] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:50:48 2022] PUT /identity/v3/projects/5f3b2a9c462542bbbd13c57d4e4e83c9/users/40284a885fd341af8e4a5274975d519e/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:48.311806 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f41388-c72a-449e-9c30-91abb289c8f6 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:48.312091 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f41388-c72a-449e-9c30-91abb289c8f6 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:48.312404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f41388-c72a-449e-9c30-91abb289c8f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:48.339546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-e1f41388-c72a-449e-9c30-91abb289c8f6 None None] MFA Rules not processed for user `40284a885fd341af8e4a5274975d519e`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:48.340765 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e1f41388-c72a-449e-9c30-91abb289c8f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:48.405769 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1458/2883] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:48 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:48.417469 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82a0332-ab31-4925-8f1b-0dfab5541692 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:48.419140 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c82a0332-ab31-4925-8f1b-0dfab5541692 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:48.484736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82a0332-ab31-4925-8f1b-0dfab5541692 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:48.488793 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cf140e-9d81-466e-99bc-668b5b75eb73 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:48.489404 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cf140e-9d81-466e-99bc-668b5b75eb73 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:48.489679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cf140e-9d81-466e-99bc-668b5b75eb73 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:48.489976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cf140e-9d81-466e-99bc-668b5b75eb73 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:48.545286 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cf140e-9d81-466e-99bc-668b5b75eb73 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:48.546742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cf140e-9d81-466e-99bc-668b5b75eb73 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:48.555466 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1426/2884] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:48 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:50.051632 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f41388-c72a-449e-9c30-91abb289c8f6 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:50.053105 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e1f41388-c72a-449e-9c30-91abb289c8f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:50.064796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f41388-c72a-449e-9c30-91abb289c8f6 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:50.068368 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d57a53-be3a-44ce-843e-6c66e9cb9e92 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:50.068958 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d57a53-be3a-44ce-843e-6c66e9cb9e92 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:50.069234 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d57a53-be3a-44ce-843e-6c66e9cb9e92 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:50.069501 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d57a53-be3a-44ce-843e-6c66e9cb9e92 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:50.124912 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d57a53-be3a-44ce-843e-6c66e9cb9e92 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:50.126258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d57a53-be3a-44ce-843e-6c66e9cb9e92 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:50.135025 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1459/2885] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:50 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:50.652087 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cf140e-9d81-466e-99bc-668b5b75eb73 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:50.660643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cf140e-9d81-466e-99bc-668b5b75eb73 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:50.664268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a46a65b-22a3-416f-80b8-7da152f302c4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:50.664946 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a46a65b-22a3-416f-80b8-7da152f302c4 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:50.665224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a46a65b-22a3-416f-80b8-7da152f302c4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:50.665494 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a46a65b-22a3-416f-80b8-7da152f302c4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:50.674149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a46a65b-22a3-416f-80b8-7da152f302c4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:50.675745 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a46a65b-22a3-416f-80b8-7da152f302c4 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:50.685767 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1427/2886] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:50 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:54.558341 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d57a53-be3a-44ce-843e-6c66e9cb9e92 service cinder] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:54.607648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:54.620014 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d57a53-be3a-44ce-843e-6c66e9cb9e92 service cinder] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:54.624680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8a843e-100b-4f97-bb0d-450ea69f0beb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:54.625298 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8a843e-100b-4f97-bb0d-450ea69f0beb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:54.625579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8a843e-100b-4f97-bb0d-450ea69f0beb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:54.625851 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8a843e-100b-4f97-bb0d-450ea69f0beb admin admin] PATH_INFO: `/v3/users/b8a9db6d79774f75b6fb8a4843c4ee27` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:54.626954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8a843e-100b-4f97-bb0d-450ea69f0beb admin admin] RBAC: Authorizing `identity:delete_user(user_id=b8a9db6d79774f75b6fb8a4843c4ee27)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:54.628964 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8a843e-100b-4f97-bb0d-450ea69f0beb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:54.673906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-cd8a843e-100b-4f97-bb0d-450ea69f0beb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b8a9db6d79774f75b6fb8a4843c4ee27'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:54.674555 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-cd8a843e-100b-4f97-bb0d-450ea69f0beb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b8a9db6d79774f75b6fb8a4843c4ee27'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:54.681984 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-cd8a843e-100b-4f97-bb0d-450ea69f0beb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b8a9db6d79774f75b6fb8a4843c4ee27'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:54.682602 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-cd8a843e-100b-4f97-bb0d-450ea69f0beb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b8a9db6d79774f75b6fb8a4843c4ee27'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:54.694176 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-cd8a843e-100b-4f97-bb0d-450ea69f0beb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b8a9db6d79774f75b6fb8a4843c4ee27'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:54.706523 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1460/2887] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:50:54 2022] DELETE /identity/v3/users/b8a9db6d79774f75b6fb8a4843c4ee27 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:54.711175 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1fe37e-98fd-4b2b-b39f-6824097abf05 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:54.711710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1fe37e-98fd-4b2b-b39f-6824097abf05 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:54.711710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1fe37e-98fd-4b2b-b39f-6824097abf05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:54.730983 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3e1fe37e-98fd-4b2b-b39f-6824097abf05 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:54.732238 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3e1fe37e-98fd-4b2b-b39f-6824097abf05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:54.751475 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:54.803673 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1428/2888] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:54.814067 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8a843e-100b-4f97-bb0d-450ea69f0beb admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:54.815473 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-cd8a843e-100b-4f97-bb0d-450ea69f0beb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:54.871720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8a843e-100b-4f97-bb0d-450ea69f0beb admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:54.875278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef1147f-7818-46ce-850e-079ddcce5184 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:54.875871 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef1147f-7818-46ce-850e-079ddcce5184 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:54.876146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef1147f-7818-46ce-850e-079ddcce5184 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:54.876457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef1147f-7818-46ce-850e-079ddcce5184 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:54.885756 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef1147f-7818-46ce-850e-079ddcce5184 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:54.887162 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef1147f-7818-46ce-850e-079ddcce5184 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:54.896397 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1461/2889] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:55.223948 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1fe37e-98fd-4b2b-b39f-6824097abf05 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:55.225882 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3e1fe37e-98fd-4b2b-b39f-6824097abf05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:55.238456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1fe37e-98fd-4b2b-b39f-6824097abf05 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:55.246252 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:55.246976 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:55.252519 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:55.253110 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] PATH_INFO: `/v3/projects/3df28f392e2842b496fe6ea6c0614103` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:55.254875 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3df28f392e2842b496fe6ea6c0614103)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:55.257491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:55.312415 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3df28f392e2842b496fe6ea6c0614103', 'request_id': 'req-2d125f56-c116-4342-a98a-3f34a7084ee1'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.318755 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3df28f392e2842b496fe6ea6c0614103', 'request_id': 'req-2d125f56-c116-4342-a98a-3f34a7084ee1'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.326997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3df28f392e2842b496fe6ea6c0614103', 'request_id': 'req-2d125f56-c116-4342-a98a-3f34a7084ee1'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.337735 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] The token cache is being invalidate because project 3df28f392e2842b496fe6ea6c0614103 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:50:55.338016 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.338634 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.340102 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1429/2890] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:50:55 2022] DELETE /identity/v3/projects/3df28f392e2842b496fe6ea6c0614103 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:55.344022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef1147f-7818-46ce-850e-079ddcce5184 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:55.353386 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef1147f-7818-46ce-850e-079ddcce5184 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:55.409146 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da787db8-1026-49b7-8838-ccc11e53c893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:55.409753 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da787db8-1026-49b7-8838-ccc11e53c893 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:55.410027 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da787db8-1026-49b7-8838-ccc11e53c893 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:55.410297 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da787db8-1026-49b7-8838-ccc11e53c893 admin admin] PATH_INFO: `/v3/users/22ad3dcc8adb4cfabe2f05a873454798` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:55.411398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da787db8-1026-49b7-8838-ccc11e53c893 admin admin] RBAC: Authorizing `identity:delete_user(user_id=22ad3dcc8adb4cfabe2f05a873454798)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:55.413906 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da787db8-1026-49b7-8838-ccc11e53c893 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:55.427288 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:55.462581 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-da787db8-1026-49b7-8838-ccc11e53c893 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '22ad3dcc8adb4cfabe2f05a873454798'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.463166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-da787db8-1026-49b7-8838-ccc11e53c893 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '22ad3dcc8adb4cfabe2f05a873454798'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.470860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-da787db8-1026-49b7-8838-ccc11e53c893 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '22ad3dcc8adb4cfabe2f05a873454798'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.471502 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-da787db8-1026-49b7-8838-ccc11e53c893 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '22ad3dcc8adb4cfabe2f05a873454798'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.483319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-da787db8-1026-49b7-8838-ccc11e53c893 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '22ad3dcc8adb4cfabe2f05a873454798'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.496460 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1462/2891] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:50:55 2022] DELETE /identity/v3/users/22ad3dcc8adb4cfabe2f05a873454798 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:55.765832 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:55.767286 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:55.783494 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d125f56-c116-4342-a98a-3f34a7084ee1 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:55.841482 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:55.842139 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:55.842421 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:55.842699 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] PATH_INFO: `/v3/projects/4abd488c1c394efbab6b06224b067fc0` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:55.843826 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4abd488c1c394efbab6b06224b067fc0)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:55.845772 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:55.890845 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4abd488c1c394efbab6b06224b067fc0', 'request_id': 'req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.892097 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:55.897092 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4abd488c1c394efbab6b06224b067fc0', 'request_id': 'req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.905209 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4abd488c1c394efbab6b06224b067fc0', 'request_id': 'req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.916501 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] The token cache is being invalidate because project 4abd488c1c394efbab6b06224b067fc0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:50:55.916786 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.917400 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:55.918853 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1430/2892] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:50:55 2022] DELETE /identity/v3/projects/4abd488c1c394efbab6b06224b067fc0 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:55.922946 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da787db8-1026-49b7-8838-ccc11e53c893 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:55.924357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-da787db8-1026-49b7-8838-ccc11e53c893 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:55.937120 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da787db8-1026-49b7-8838-ccc11e53c893 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:55.996190 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c812e08e-8600-4a5d-bc81-012219808c76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:55.996788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c812e08e-8600-4a5d-bc81-012219808c76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:55.997068 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c812e08e-8600-4a5d-bc81-012219808c76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:55.997347 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c812e08e-8600-4a5d-bc81-012219808c76 admin admin] PATH_INFO: `/v3/users/40284a885fd341af8e4a5274975d519e` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:55.998469 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c812e08e-8600-4a5d-bc81-012219808c76 admin admin] RBAC: Authorizing `identity:delete_user(user_id=40284a885fd341af8e4a5274975d519e)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:56.000399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c812e08e-8600-4a5d-bc81-012219808c76 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:56.047616 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c812e08e-8600-4a5d-bc81-012219808c76 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '40284a885fd341af8e4a5274975d519e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:56.048270 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c812e08e-8600-4a5d-bc81-012219808c76 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '40284a885fd341af8e4a5274975d519e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:56.055640 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c812e08e-8600-4a5d-bc81-012219808c76 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '40284a885fd341af8e4a5274975d519e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:56.056284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c812e08e-8600-4a5d-bc81-012219808c76 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '40284a885fd341af8e4a5274975d519e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:56.067680 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-c812e08e-8600-4a5d-bc81-012219808c76 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '40284a885fd341af8e4a5274975d519e'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:56.080459 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1463/2893] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:50:55 2022] DELETE /identity/v3/users/40284a885fd341af8e4a5274975d519e => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:56.377091 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:56.378512 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:56.392159 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a118f9-dfdb-439a-9ce8-98cea6b0bea5 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:56.456431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902feb54-dfdf-4f28-8dfd-0340ab84d25a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:56.456431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902feb54-dfdf-4f28-8dfd-0340ab84d25a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:56.456431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902feb54-dfdf-4f28-8dfd-0340ab84d25a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:56.456431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902feb54-dfdf-4f28-8dfd-0340ab84d25a admin admin] PATH_INFO: `/v3/projects/5f3b2a9c462542bbbd13c57d4e4e83c9` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:56.457544 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902feb54-dfdf-4f28-8dfd-0340ab84d25a admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f3b2a9c462542bbbd13c57d4e4e83c9)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:56.459518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902feb54-dfdf-4f28-8dfd-0340ab84d25a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:56.510224 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-902feb54-dfdf-4f28-8dfd-0340ab84d25a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f3b2a9c462542bbbd13c57d4e4e83c9', 'request_id': 'req-902feb54-dfdf-4f28-8dfd-0340ab84d25a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:56.516004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-902feb54-dfdf-4f28-8dfd-0340ab84d25a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f3b2a9c462542bbbd13c57d4e4e83c9', 'request_id': 'req-902feb54-dfdf-4f28-8dfd-0340ab84d25a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:56.525042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-902feb54-dfdf-4f28-8dfd-0340ab84d25a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f3b2a9c462542bbbd13c57d4e4e83c9', 'request_id': 'req-902feb54-dfdf-4f28-8dfd-0340ab84d25a'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:56.536630 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-902feb54-dfdf-4f28-8dfd-0340ab84d25a admin admin] The token cache is being invalidate because project 5f3b2a9c462542bbbd13c57d4e4e83c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:50:56.536914 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-902feb54-dfdf-4f28-8dfd-0340ab84d25a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:56.537557 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-902feb54-dfdf-4f28-8dfd-0340ab84d25a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:50:56.539168 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1431/2894] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:50:56 2022] DELETE /identity/v3/projects/5f3b2a9c462542bbbd13c57d4e4e83c9 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:56.544040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4583ee5-5df7-4224-802f-a77b70a50507 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:56.544402 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4583ee5-5df7-4224-802f-a77b70a50507 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:56.544686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4583ee5-5df7-4224-802f-a77b70a50507 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:56.563930 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-e4583ee5-5df7-4224-802f-a77b70a50507 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:56.565175 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e4583ee5-5df7-4224-802f-a77b70a50507 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:56.571281 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:56.628793 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1464/2895] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:56.636477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f574e0d6-6b25-46fd-a9b5-89eb0f65f1a3 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:56.636757 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f574e0d6-6b25-46fd-a9b5-89eb0f65f1a3 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:56.637027 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f574e0d6-6b25-46fd-a9b5-89eb0f65f1a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:56.650431 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4583ee5-5df7-4224-802f-a77b70a50507 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:56.651813 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-e4583ee5-5df7-4224-802f-a77b70a50507 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:56.656278 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-f574e0d6-6b25-46fd-a9b5-89eb0f65f1a3 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:56.657471 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f574e0d6-6b25-46fd-a9b5-89eb0f65f1a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:56.667384 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1432/2896] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:56.671465 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f574e0d6-6b25-46fd-a9b5-89eb0f65f1a3 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:56.681776 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f574e0d6-6b25-46fd-a9b5-89eb0f65f1a3 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:56.686657 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf310bf-f545-442b-ba91-697cff2ed29f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:56.687303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf310bf-f545-442b-ba91-697cff2ed29f admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:56.687577 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf310bf-f545-442b-ba91-697cff2ed29f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:56.687843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf310bf-f545-442b-ba91-697cff2ed29f admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:56.688502 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf310bf-f545-442b-ba91-697cff2ed29f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:50:56.688801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf310bf-f545-442b-ba91-697cff2ed29f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:56.690370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf310bf-f545-442b-ba91-697cff2ed29f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:56.705285 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1433/2897] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:50:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:50:56.706985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf310bf-f545-442b-ba91-697cff2ed29f admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:56.711009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4583ee5-5df7-4224-802f-a77b70a50507 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:56.714720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1409b97a-145e-46fd-8a23-37f4eb07a06d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:56.715329 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1409b97a-145e-46fd-8a23-37f4eb07a06d service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:56.715611 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1409b97a-145e-46fd-8a23-37f4eb07a06d service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:56.715888 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1409b97a-145e-46fd-8a23-37f4eb07a06d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:56.716064 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf310bf-f545-442b-ba91-697cff2ed29f admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:56.717658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1409b97a-145e-46fd-8a23-37f4eb07a06d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:56.720880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7759729b-bcdd-44ad-9fb0-efc87be35451 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:56.721450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7759729b-bcdd-44ad-9fb0-efc87be35451 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:56.721728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7759729b-bcdd-44ad-9fb0-efc87be35451 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:56.721999 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7759729b-bcdd-44ad-9fb0-efc87be35451 admin admin] PATH_INFO: `/v3/projects` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:56.722619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7759729b-bcdd-44ad-9fb0-efc87be35451 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:56.724525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7759729b-bcdd-44ad-9fb0-efc87be35451 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:56.755549 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1434/2898] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:50:56 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:56.756619 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7759729b-bcdd-44ad-9fb0-efc87be35451 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:56.782213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1409b97a-145e-46fd-8a23-37f4eb07a06d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:56.783780 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1409b97a-145e-46fd-8a23-37f4eb07a06d service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:56.785987 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:DELETE=10,keystone:SELECT=87,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:56.795863 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1465/2899] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:56 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:50:56.820302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7759729b-bcdd-44ad-9fb0-efc87be35451 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:56.825112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60eb7ac-80dc-4d96-991a-4b162b29b9d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:56.825703 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60eb7ac-80dc-4d96-991a-4b162b29b9d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:56.825978 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60eb7ac-80dc-4d96-991a-4b162b29b9d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:56.826253 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60eb7ac-80dc-4d96-991a-4b162b29b9d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:56.826876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60eb7ac-80dc-4d96-991a-4b162b29b9d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:56.828862 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60eb7ac-80dc-4d96-991a-4b162b29b9d4 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:56.855050 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1435/2900] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:50:56 2022] POST /identity/v3/users => generated 292 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:56.856380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1409b97a-145e-46fd-8a23-37f4eb07a06d service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:56.867982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1409b97a-145e-46fd-8a23-37f4eb07a06d service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:56.872694 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f247cc-bdce-44b2-84cd-4de81d5a02d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:56.873311 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f247cc-bdce-44b2-84cd-4de81d5a02d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:56.873586 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f247cc-bdce-44b2-84cd-4de81d5a02d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:56.873855 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f247cc-bdce-44b2-84cd-4de81d5a02d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:56.874500 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f247cc-bdce-44b2-84cd-4de81d5a02d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:56.876069 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f247cc-bdce-44b2-84cd-4de81d5a02d9 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:56.886802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1466/2901] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:56.890993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60eb7ac-80dc-4d96-991a-4b162b29b9d4 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:56.901289 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60eb7ac-80dc-4d96-991a-4b162b29b9d4 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:56.905961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b25e22c-091a-4f86-829d-ae358b02e802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:56.906558 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b25e22c-091a-4f86-829d-ae358b02e802 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:56.906836 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b25e22c-091a-4f86-829d-ae358b02e802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:56.907107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b25e22c-091a-4f86-829d-ae358b02e802 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:56.907745 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b25e22c-091a-4f86-829d-ae358b02e802 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:56.909381 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b25e22c-091a-4f86-829d-ae358b02e802 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:56.919212 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1436/2902] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:56 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:56.923527 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f247cc-bdce-44b2-84cd-4de81d5a02d9 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:56.936357 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f247cc-bdce-44b2-84cd-4de81d5a02d9 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:56.941063 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8047039f-344f-43a2-afe1-53cb40e800e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:56.941774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8047039f-344f-43a2-afe1-53cb40e800e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:56.942053 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8047039f-344f-43a2-afe1-53cb40e800e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:56.942323 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8047039f-344f-43a2-afe1-53cb40e800e6 admin admin] PATH_INFO: `/v3/projects/9f776286fdb94e37a26d496f2d60cff7/users/d8f1350219714baaabac786d7785ef24/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:56.958658 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8047039f-344f-43a2-afe1-53cb40e800e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f776286fdb94e37a26d496f2d60cff7, user_id=d8f1350219714baaabac786d7785ef24, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:56.962065 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8047039f-344f-43a2-afe1-53cb40e800e6 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:56.975323 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1467/2903] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:50:56 2022] PUT /identity/v3/projects/9f776286fdb94e37a26d496f2d60cff7/users/d8f1350219714baaabac786d7785ef24/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:56.987426 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffd5639-4071-417d-b5c0-667f9cf373b8 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:56.987994 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffd5639-4071-417d-b5c0-667f9cf373b8 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:56.988265 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffd5639-4071-417d-b5c0-667f9cf373b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:57.016373 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-4ffd5639-4071-417d-b5c0-667f9cf373b8 None None] MFA Rules not processed for user `d8f1350219714baaabac786d7785ef24`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:57.017578 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-4ffd5639-4071-417d-b5c0-667f9cf373b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:57.083867 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1437/2904] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:56 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 28 12:50:57.088433 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8047039f-344f-43a2-afe1-53cb40e800e6 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:57.119663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:57.152091 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8047039f-344f-43a2-afe1-53cb40e800e6 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:57.156870 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e0612f-3b15-4c48-aa91-107526fb9e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:57.157471 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e0612f-3b15-4c48-aa91-107526fb9e5e admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:57.157745 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e0612f-3b15-4c48-aa91-107526fb9e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:57.158030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e0612f-3b15-4c48-aa91-107526fb9e5e admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:57.158662 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e0612f-3b15-4c48-aa91-107526fb9e5e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:57.160567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e0612f-3b15-4c48-aa91-107526fb9e5e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:57.188194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1468/2905] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:50:57 2022] POST /identity/v3/projects => generated 383 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:57.192286 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffd5639-4071-417d-b5c0-667f9cf373b8 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:57.205506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffd5639-4071-417d-b5c0-667f9cf373b8 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:57.263483 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ea8056-6881-47db-868e-f663cb43e324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:57.264083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ea8056-6881-47db-868e-f663cb43e324 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:57.264393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ea8056-6881-47db-868e-f663cb43e324 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:57.264665 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ea8056-6881-47db-868e-f663cb43e324 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:57.265329 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ea8056-6881-47db-868e-f663cb43e324 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:57.267370 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ea8056-6881-47db-868e-f663cb43e324 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:57.289302 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1438/2906] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:50:57 2022] POST /identity/v3/users => generated 292 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:50:57.293287 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e0612f-3b15-4c48-aa91-107526fb9e5e admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:57.303676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e0612f-3b15-4c48-aa91-107526fb9e5e admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:57.308873 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a08cd7-155a-4129-a374-690d1dc3444c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:57.309410 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a08cd7-155a-4129-a374-690d1dc3444c admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:57.309695 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a08cd7-155a-4129-a374-690d1dc3444c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:57.309969 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a08cd7-155a-4129-a374-690d1dc3444c admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:57.310610 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a08cd7-155a-4129-a374-690d1dc3444c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:57.312292 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a08cd7-155a-4129-a374-690d1dc3444c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:57.322994 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1469/2907] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:57.327176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ea8056-6881-47db-868e-f663cb43e324 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:57.337616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ea8056-6881-47db-868e-f663cb43e324 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:57.342148 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978bd74d-3092-4ff2-9b14-182638bd2563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:57.342729 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978bd74d-3092-4ff2-9b14-182638bd2563 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:57.343004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978bd74d-3092-4ff2-9b14-182638bd2563 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:57.343269 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978bd74d-3092-4ff2-9b14-182638bd2563 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:57.343931 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978bd74d-3092-4ff2-9b14-182638bd2563 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:57.345584 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978bd74d-3092-4ff2-9b14-182638bd2563 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:57.355915 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1439/2908] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:57 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:50:57.360045 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a08cd7-155a-4129-a374-690d1dc3444c admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:57.370559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a08cd7-155a-4129-a374-690d1dc3444c admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:57.375189 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecb4789-68f7-478f-81ff-000a58e6ff02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:57.375831 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecb4789-68f7-478f-81ff-000a58e6ff02 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:57.376108 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecb4789-68f7-478f-81ff-000a58e6ff02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:57.376408 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecb4789-68f7-478f-81ff-000a58e6ff02 admin admin] PATH_INFO: `/v3/projects/70232843d86d4842a8074e5514a3dec8/users/4cd136e6ebfa46b2991e2482e04a39ce/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:57.392128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecb4789-68f7-478f-81ff-000a58e6ff02 admin admin] RBAC: Authorizing `identity:create_grant(project_id=70232843d86d4842a8074e5514a3dec8, user_id=4cd136e6ebfa46b2991e2482e04a39ce, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:57.394777 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecb4789-68f7-478f-81ff-000a58e6ff02 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:57.408370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1470/2909] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:50:57 2022] PUT /identity/v3/projects/70232843d86d4842a8074e5514a3dec8/users/4cd136e6ebfa46b2991e2482e04a39ce/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:57.418380 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfecf8c-8045-46d1-8061-7e72ca66758b None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:57.418731 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfecf8c-8045-46d1-8061-7e72ca66758b None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:57.419125 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfecf8c-8045-46d1-8061-7e72ca66758b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:57.445563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:57.449647 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-fbfecf8c-8045-46d1-8061-7e72ca66758b None None] MFA Rules not processed for user `4cd136e6ebfa46b2991e2482e04a39ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:57.451511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-fbfecf8c-8045-46d1-8061-7e72ca66758b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:57.524530 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1440/2910] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:57.529017 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecb4789-68f7-478f-81ff-000a58e6ff02 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:57.590078 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecb4789-68f7-478f-81ff-000a58e6ff02 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:57.594796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e4e24d-4090-4e1f-9b6f-52804ea3b16e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:57.595390 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e4e24d-4090-4e1f-9b6f-52804ea3b16e admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:57.595664 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e4e24d-4090-4e1f-9b6f-52804ea3b16e admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:57.595979 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e4e24d-4090-4e1f-9b6f-52804ea3b16e admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:57.596651 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e4e24d-4090-4e1f-9b6f-52804ea3b16e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:57.598693 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e4e24d-4090-4e1f-9b6f-52804ea3b16e admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:57.629206 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1471/2911] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:50:57 2022] POST /identity/v3/projects => generated 385 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:50:57.630233 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfecf8c-8045-46d1-8061-7e72ca66758b None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:57.641665 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfecf8c-8045-46d1-8061-7e72ca66758b None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:57.701895 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8925dd93-8e65-46d1-9710-9be499759b14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:57.702498 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8925dd93-8e65-46d1-9710-9be499759b14 admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:57.702773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8925dd93-8e65-46d1-9710-9be499759b14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:57.703042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8925dd93-8e65-46d1-9710-9be499759b14 admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:57.703679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8925dd93-8e65-46d1-9710-9be499759b14 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:57.705663 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8925dd93-8e65-46d1-9710-9be499759b14 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:57.729200 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1441/2912] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:50:57 2022] POST /identity/v3/users => generated 299 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:50:57.733413 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e4e24d-4090-4e1f-9b6f-52804ea3b16e admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:57.744701 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e4e24d-4090-4e1f-9b6f-52804ea3b16e admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:57.749391 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a0c2d6-30df-449e-8197-0de8661e8da8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:57.749981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a0c2d6-30df-449e-8197-0de8661e8da8 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:57.750255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a0c2d6-30df-449e-8197-0de8661e8da8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:57.750525 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a0c2d6-30df-449e-8197-0de8661e8da8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:57.751165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a0c2d6-30df-449e-8197-0de8661e8da8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:57.752773 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a0c2d6-30df-449e-8197-0de8661e8da8 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:57.763151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1472/2913] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 28 12:50:57.767422 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8925dd93-8e65-46d1-9710-9be499759b14 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:57.778174 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8925dd93-8e65-46d1-9710-9be499759b14 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:57.783075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e96bfc3-8c4c-4056-aa98-18d53c412a5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:57.783754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e96bfc3-8c4c-4056-aa98-18d53c412a5a admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:57.784033 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e96bfc3-8c4c-4056-aa98-18d53c412a5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:57.784328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e96bfc3-8c4c-4056-aa98-18d53c412a5a admin admin] PATH_INFO: `/v3/projects/8b72e0cd44a1429a87b142d0e4f40ea1/users/6c7c19578881498bb6bfec85a340c9a4/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:57.800010 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e96bfc3-8c4c-4056-aa98-18d53c412a5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b72e0cd44a1429a87b142d0e4f40ea1, user_id=6c7c19578881498bb6bfec85a340c9a4, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:57.802797 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e96bfc3-8c4c-4056-aa98-18d53c412a5a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:57.817249 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1442/2914] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:50:57 2022] PUT /identity/v3/projects/8b72e0cd44a1429a87b142d0e4f40ea1/users/6c7c19578881498bb6bfec85a340c9a4/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:57.821075 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a0c2d6-30df-449e-8197-0de8661e8da8 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:57.831373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a0c2d6-30df-449e-8197-0de8661e8da8 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:57.880128 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:57.891576 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d35307a-4752-47c7-8630-8b13c77751e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:57.892181 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d35307a-4752-47c7-8630-8b13c77751e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:57.892489 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d35307a-4752-47c7-8630-8b13c77751e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:57.892791 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d35307a-4752-47c7-8630-8b13c77751e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:57.893432 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d35307a-4752-47c7-8630-8b13c77751e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:57.895054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d35307a-4752-47c7-8630-8b13c77751e8 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:57.905826 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1473/2915] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:50:57 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:50:57.910112 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e96bfc3-8c4c-4056-aa98-18d53c412a5a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:57.921403 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e96bfc3-8c4c-4056-aa98-18d53c412a5a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:57.926843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59414444-0aa6-4d53-8044-e07f9f4f644a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:57.927500 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59414444-0aa6-4d53-8044-e07f9f4f644a admin admin] REQUEST_METHOD: `PUT` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:57.927780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59414444-0aa6-4d53-8044-e07f9f4f644a admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:57.928052 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59414444-0aa6-4d53-8044-e07f9f4f644a admin admin] PATH_INFO: `/v3/projects/8b72e0cd44a1429a87b142d0e4f40ea1/users/6c7c19578881498bb6bfec85a340c9a4/roles/f80bb7fe38494b9aabf015d2ff6fa215` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:57.930707 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59414444-0aa6-4d53-8044-e07f9f4f644a admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b72e0cd44a1429a87b142d0e4f40ea1, user_id=6c7c19578881498bb6bfec85a340c9a4, role_id=f80bb7fe38494b9aabf015d2ff6fa215)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:57.933393 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59414444-0aa6-4d53-8044-e07f9f4f644a admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:57.944093 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1443/2916] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:50:57 2022] PUT /identity/v3/projects/8b72e0cd44a1429a87b142d0e4f40ea1/users/6c7c19578881498bb6bfec85a340c9a4/roles/f80bb7fe38494b9aabf015d2ff6fa215 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:50:57.954183 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049042a4-223b-4ba9-aa98-0983271228bf None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:57.954667 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049042a4-223b-4ba9-aa98-0983271228bf None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:57.954761 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049042a4-223b-4ba9-aa98-0983271228bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:57.983742 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-049042a4-223b-4ba9-aa98-0983271228bf None None] MFA Rules not processed for user `6c7c19578881498bb6bfec85a340c9a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:50:57.985096 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-049042a4-223b-4ba9-aa98-0983271228bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:58.055569 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1474/2917] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:50:57 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:50:58.066539 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59414444-0aa6-4d53-8044-e07f9f4f644a admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:58.068297 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-59414444-0aa6-4d53-8044-e07f9f4f644a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:58.127616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59414444-0aa6-4d53-8044-e07f9f4f644a admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:58.131170 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafaccca-5a7e-4bea-a8d3-dd13dd92aa0d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:58.131801 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafaccca-5a7e-4bea-a8d3-dd13dd92aa0d service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:58.132083 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafaccca-5a7e-4bea-a8d3-dd13dd92aa0d service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:58.132381 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafaccca-5a7e-4bea-a8d3-dd13dd92aa0d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:58.140646 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:50:58.190611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafaccca-5a7e-4bea-a8d3-dd13dd92aa0d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:58.192135 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafaccca-5a7e-4bea-a8d3-dd13dd92aa0d service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:58.200891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1444/2918] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:58.654663 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049042a4-223b-4ba9-aa98-0983271228bf None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:58.656235 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-049042a4-223b-4ba9-aa98-0983271228bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:58.741802 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049042a4-223b-4ba9-aa98-0983271228bf None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:58.747302 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca595945-3b28-4785-9ba5-784faf1b709b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:58.747914 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca595945-3b28-4785-9ba5-784faf1b709b service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:58.748364 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca595945-3b28-4785-9ba5-784faf1b709b service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:58.748647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca595945-3b28-4785-9ba5-784faf1b709b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:58.750397 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-ca595945-3b28-4785-9ba5-784faf1b709b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:58.763294 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca595945-3b28-4785-9ba5-784faf1b709b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:58.764910 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca595945-3b28-4785-9ba5-784faf1b709b service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:58.775885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1475/2919] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:50:59.742827 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafaccca-5a7e-4bea-a8d3-dd13dd92aa0d service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:50:59.744437 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-dafaccca-5a7e-4bea-a8d3-dd13dd92aa0d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:50:59.756060 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafaccca-5a7e-4bea-a8d3-dd13dd92aa0d service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:50:59.759710 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5576f2fb-ee9f-4e8c-bdaf-b20e230d0774 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:50:59.760342 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5576f2fb-ee9f-4e8c-bdaf-b20e230d0774 service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:50:59.760632 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5576f2fb-ee9f-4e8c-bdaf-b20e230d0774 service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:50:59.760898 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5576f2fb-ee9f-4e8c-bdaf-b20e230d0774 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:50:59.822394 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5576f2fb-ee9f-4e8c-bdaf-b20e230d0774 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:50:59.823847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5576f2fb-ee9f-4e8c-bdaf-b20e230d0774 service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:50:59.833312 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1445/2920] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:50:59 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:51:00.349550 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca595945-3b28-4785-9ba5-784faf1b709b service nova] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:00.358774 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca595945-3b28-4785-9ba5-784faf1b709b service nova] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:00.362320 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf07b44-9ab4-4d61-82de-4ff723dccd6c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:00.363156 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf07b44-9ab4-4d61-82de-4ff723dccd6c service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:00.363436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf07b44-9ab4-4d61-82de-4ff723dccd6c service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:00.363711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf07b44-9ab4-4d61-82de-4ff723dccd6c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:00.373065 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf07b44-9ab4-4d61-82de-4ff723dccd6c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:00.374471 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf07b44-9ab4-4d61-82de-4ff723dccd6c service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:00.384404 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1476/2921] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:51:00 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 28 12:51:04.231728 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5576f2fb-ee9f-4e8c-bdaf-b20e230d0774 service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:04.302511 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5576f2fb-ee9f-4e8c-bdaf-b20e230d0774 service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:04.307029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e25ff7d-6fe0-45bd-a49e-a9321dd92081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:04.307643 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e25ff7d-6fe0-45bd-a49e-a9321dd92081 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:04.307920 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e25ff7d-6fe0-45bd-a49e-a9321dd92081 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:04.308188 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e25ff7d-6fe0-45bd-a49e-a9321dd92081 admin admin] PATH_INFO: `/v3/users/d8f1350219714baaabac786d7785ef24` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:04.309328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e25ff7d-6fe0-45bd-a49e-a9321dd92081 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8f1350219714baaabac786d7785ef24)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:04.311240 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e25ff7d-6fe0-45bd-a49e-a9321dd92081 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:04.357764 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0e25ff7d-6fe0-45bd-a49e-a9321dd92081 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8f1350219714baaabac786d7785ef24'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:04.358408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0e25ff7d-6fe0-45bd-a49e-a9321dd92081 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8f1350219714baaabac786d7785ef24'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:04.358998 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0e25ff7d-6fe0-45bd-a49e-a9321dd92081 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8f1350219714baaabac786d7785ef24'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:04.361012 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:51:04.367704 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0e25ff7d-6fe0-45bd-a49e-a9321dd92081 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8f1350219714baaabac786d7785ef24'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:04.379443 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0e25ff7d-6fe0-45bd-a49e-a9321dd92081 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8f1350219714baaabac786d7785ef24'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:04.392149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1446/2922] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:51:04 2022] DELETE /identity/v3/users/d8f1350219714baaabac786d7785ef24 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:51:04.396620 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d38902-a363-4cfb-90de-6b82d1d6be06 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:04.397098 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d38902-a363-4cfb-90de-6b82d1d6be06 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:04.397803 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d38902-a363-4cfb-90de-6b82d1d6be06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:04.417860 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-97d38902-a363-4cfb-90de-6b82d1d6be06 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:51:04.419100 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-97d38902-a363-4cfb-90de-6b82d1d6be06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:51:04.460834 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=100 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:51:04.479197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1477/2923] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:51:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:51:04.489229 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e25ff7d-6fe0-45bd-a49e-a9321dd92081 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:04.490625 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0e25ff7d-6fe0-45bd-a49e-a9321dd92081 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:51:04.546932 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e25ff7d-6fe0-45bd-a49e-a9321dd92081 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:04.550560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73797152-f457-4629-a22d-7feb52e24274 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:04.551147 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73797152-f457-4629-a22d-7feb52e24274 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:04.551423 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73797152-f457-4629-a22d-7feb52e24274 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:04.551689 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73797152-f457-4629-a22d-7feb52e24274 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:04.560894 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73797152-f457-4629-a22d-7feb52e24274 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:04.563431 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73797152-f457-4629-a22d-7feb52e24274 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:04.575491 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1447/2924] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:51:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:51:04.840022 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d38902-a363-4cfb-90de-6b82d1d6be06 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:04.841509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-97d38902-a363-4cfb-90de-6b82d1d6be06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:51:04.853258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d38902-a363-4cfb-90de-6b82d1d6be06 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:04.857980 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:04.858618 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:04.858896 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:04.859166 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] PATH_INFO: `/v3/projects/9f776286fdb94e37a26d496f2d60cff7` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:04.860293 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f776286fdb94e37a26d496f2d60cff7)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:04.862488 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:04.911113 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f776286fdb94e37a26d496f2d60cff7', 'request_id': 'req-32149b40-f61f-4bfc-909f-794ce2f40173'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:04.916622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f776286fdb94e37a26d496f2d60cff7', 'request_id': 'req-32149b40-f61f-4bfc-909f-794ce2f40173'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:04.923886 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f776286fdb94e37a26d496f2d60cff7', 'request_id': 'req-32149b40-f61f-4bfc-909f-794ce2f40173'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:04.935499 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] The token cache is being invalidate because project 9f776286fdb94e37a26d496f2d60cff7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:51:04.935787 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:04.936467 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:04.937937 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1478/2925] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:51:04 2022] DELETE /identity/v3/projects/9f776286fdb94e37a26d496f2d60cff7 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:51:04.941928 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73797152-f457-4629-a22d-7feb52e24274 service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:04.952073 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73797152-f457-4629-a22d-7feb52e24274 service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:05.008557 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a0abb2-0efe-49ad-afc2-af7afe4dda0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:05.009129 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0abb2-0efe-49ad-afc2-af7afe4dda0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:05.009409 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0abb2-0efe-49ad-afc2-af7afe4dda0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:05.009679 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0abb2-0efe-49ad-afc2-af7afe4dda0e admin admin] PATH_INFO: `/v3/users/4cd136e6ebfa46b2991e2482e04a39ce` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:05.010791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a0abb2-0efe-49ad-afc2-af7afe4dda0e admin admin] RBAC: Authorizing `identity:delete_user(user_id=4cd136e6ebfa46b2991e2482e04a39ce)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:05.012749 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a0abb2-0efe-49ad-afc2-af7afe4dda0e admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:05.056842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-f6a0abb2-0efe-49ad-afc2-af7afe4dda0e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4cd136e6ebfa46b2991e2482e04a39ce'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.057483 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-f6a0abb2-0efe-49ad-afc2-af7afe4dda0e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4cd136e6ebfa46b2991e2482e04a39ce'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.058094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-f6a0abb2-0efe-49ad-afc2-af7afe4dda0e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4cd136e6ebfa46b2991e2482e04a39ce'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.064773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-f6a0abb2-0efe-49ad-afc2-af7afe4dda0e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4cd136e6ebfa46b2991e2482e04a39ce'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.075322 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:51:05.076691 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-f6a0abb2-0efe-49ad-afc2-af7afe4dda0e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4cd136e6ebfa46b2991e2482e04a39ce'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.091134 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1448/2926] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:51:04 2022] DELETE /identity/v3/users/4cd136e6ebfa46b2991e2482e04a39ce => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:51:05.360040 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:05.361509 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:51:05.375861 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32149b40-f61f-4bfc-909f-794ce2f40173 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:05.433010 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:05.433686 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:05.433924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:05.434193 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] PATH_INFO: `/v3/projects/70232843d86d4842a8074e5514a3dec8` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:05.435291 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] RBAC: Authorizing `identity:delete_project(project_id=70232843d86d4842a8074e5514a3dec8)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:05.437203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:05.481356 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '70232843d86d4842a8074e5514a3dec8', 'request_id': 'req-6b727c18-9435-4b15-a75f-85c53f3d4666'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.487251 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '70232843d86d4842a8074e5514a3dec8', 'request_id': 'req-6b727c18-9435-4b15-a75f-85c53f3d4666'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.496199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '70232843d86d4842a8074e5514a3dec8', 'request_id': 'req-6b727c18-9435-4b15-a75f-85c53f3d4666'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.503341 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:51:05.508453 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] The token cache is being invalidate because project 70232843d86d4842a8074e5514a3dec8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:51:05.509016 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.510095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.511869 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1479/2927] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:51:05 2022] DELETE /identity/v3/projects/70232843d86d4842a8074e5514a3dec8 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:51:05.515806 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a0abb2-0efe-49ad-afc2-af7afe4dda0e admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:05.517316 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f6a0abb2-0efe-49ad-afc2-af7afe4dda0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:51:05.529953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a0abb2-0efe-49ad-afc2-af7afe4dda0e admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:05.585773 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b712bea-ecba-484e-abc5-abbc9738f8aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:05.586325 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b712bea-ecba-484e-abc5-abbc9738f8aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:05.586605 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b712bea-ecba-484e-abc5-abbc9738f8aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:05.586878 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b712bea-ecba-484e-abc5-abbc9738f8aa admin admin] PATH_INFO: `/v3/users/6c7c19578881498bb6bfec85a340c9a4` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:05.587993 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b712bea-ecba-484e-abc5-abbc9738f8aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=6c7c19578881498bb6bfec85a340c9a4)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:05.589996 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b712bea-ecba-484e-abc5-abbc9738f8aa admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:05.636475 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5b712bea-ecba-484e-abc5-abbc9738f8aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6c7c19578881498bb6bfec85a340c9a4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.637128 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5b712bea-ecba-484e-abc5-abbc9738f8aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6c7c19578881498bb6bfec85a340c9a4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.637740 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5b712bea-ecba-484e-abc5-abbc9738f8aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6c7c19578881498bb6bfec85a340c9a4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.644520 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5b712bea-ecba-484e-abc5-abbc9738f8aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6c7c19578881498bb6bfec85a340c9a4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.656621 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-5b712bea-ecba-484e-abc5-abbc9738f8aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6c7c19578881498bb6bfec85a340c9a4'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:05.669680 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1449/2928] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:51:05 2022] DELETE /identity/v3/users/6c7c19578881498bb6bfec85a340c9a4 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:51:06.211804 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:06.213305 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:51:06.226961 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b727c18-9435-4b15-a75f-85c53f3d4666 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:06.282179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:06.282797 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:06.283076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:06.283350 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad admin admin] PATH_INFO: `/v3/projects/8b72e0cd44a1429a87b142d0e4f40ea1` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:06.284514 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad admin admin] RBAC: Authorizing `identity:delete_project(project_id=8b72e0cd44a1429a87b142d0e4f40ea1)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:06.286398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:06.332518 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8b72e0cd44a1429a87b142d0e4f40ea1', 'request_id': 'req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:06.339739 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8b72e0cd44a1429a87b142d0e4f40ea1', 'request_id': 'req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:06.347044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8b72e0cd44a1429a87b142d0e4f40ea1', 'request_id': 'req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:06.358698 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad admin admin] The token cache is being invalidate because project 8b72e0cd44a1429a87b142d0e4f40ea1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:51:06.358989 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:06.359619 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-bc9c1e76-f7f4-46b1-b209-58f3f57abaad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:06.361131 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1480/2929] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:51:06 2022] DELETE /identity/v3/projects/8b72e0cd44a1429a87b142d0e4f40ea1 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:51:06.366236 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988337dc-a391-48c2-9dc1-627742aa4df0 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:06.366518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988337dc-a391-48c2-9dc1-627742aa4df0 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:06.366791 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988337dc-a391-48c2-9dc1-627742aa4df0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:06.385810 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-988337dc-a391-48c2-9dc1-627742aa4df0 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:51:06.386985 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-988337dc-a391-48c2-9dc1-627742aa4df0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:51:06.447203 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1450/2930] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:51:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:51:06.456033 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10323352-5a32-462d-a91d-6f1918620220 None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:06.456556 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10323352-5a32-462d-a91d-6f1918620220 None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:06.456672 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10323352-5a32-462d-a91d-6f1918620220 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:06.476917 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-10323352-5a32-462d-a91d-6f1918620220 None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:51:06.478112 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-10323352-5a32-462d-a91d-6f1918620220 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:51:06.486114 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1481/2931] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:51:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:51:06.490408 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988337dc-a391-48c2-9dc1-627742aa4df0 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:06.499906 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988337dc-a391-48c2-9dc1-627742aa4df0 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:06.504489 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d8c428-5b3a-40e4-85d3-b729728c3dd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:06.505079 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d8c428-5b3a-40e4-85d3-b729728c3dd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:06.505353 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d8c428-5b3a-40e4-85d3-b729728c3dd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:06.505622 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d8c428-5b3a-40e4-85d3-b729728c3dd6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:06.506261 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d8c428-5b3a-40e4-85d3-b729728c3dd6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81592) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 28 12:51:06.506563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d8c428-5b3a-40e4-85d3-b729728c3dd6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:06.508143 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d8c428-5b3a-40e4-85d3-b729728c3dd6 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:06.510445 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:51:06.524620 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1451/2932] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Sep 28 12:51:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 28 12:51:06.525690 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10323352-5a32-462d-a91d-6f1918620220 None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:06.534799 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10323352-5a32-462d-a91d-6f1918620220 None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:06.540044 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4f4e81-8f07-43e4-a064-c3de2356f9a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:06.540698 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4f4e81-8f07-43e4-a064-c3de2356f9a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:06.540982 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4f4e81-8f07-43e4-a064-c3de2356f9a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:06.541255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4f4e81-8f07-43e4-a064-c3de2356f9a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:06.541895 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4f4e81-8f07-43e4-a064-c3de2356f9a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:06.543788 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4f4e81-8f07-43e4-a064-c3de2356f9a2 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:06.570767 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1482/2933] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Sep 28 12:51:06 2022] POST /identity/v3/projects => generated 373 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 28 12:51:06.574883 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d8c428-5b3a-40e4-85d3-b729728c3dd6 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:06.584559 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d8c428-5b3a-40e4-85d3-b729728c3dd6 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:06.647678 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cb1cf1-216f-459b-8243-f51e4cf8d7be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:06.648223 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cb1cf1-216f-459b-8243-f51e4cf8d7be admin admin] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:06.648532 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cb1cf1-216f-459b-8243-f51e4cf8d7be admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:06.648803 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cb1cf1-216f-459b-8243-f51e4cf8d7be admin admin] PATH_INFO: `/v3/users` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:06.649425 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cb1cf1-216f-459b-8243-f51e4cf8d7be admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:06.651337 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cb1cf1-216f-459b-8243-f51e4cf8d7be admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:06.671802 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1452/2934] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Sep 28 12:51:06 2022] POST /identity/v3/users => generated 287 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 28 12:51:06.675924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4f4e81-8f07-43e4-a064-c3de2356f9a2 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:06.686076 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4f4e81-8f07-43e4-a064-c3de2356f9a2 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:06.690783 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313cbd6-a9a1-4c03-9c4a-eefb6a03ebcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:06.691373 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313cbd6-a9a1-4c03-9c4a-eefb6a03ebcc admin admin] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:06.691657 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313cbd6-a9a1-4c03-9c4a-eefb6a03ebcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:06.691932 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313cbd6-a9a1-4c03-9c4a-eefb6a03ebcc admin admin] PATH_INFO: `/v3/roles` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:06.692734 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e313cbd6-a9a1-4c03-9c4a-eefb6a03ebcc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:06.694925 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e313cbd6-a9a1-4c03-9c4a-eefb6a03ebcc admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:06.704579 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1483/2935] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:51:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:51:06.708361 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cb1cf1-216f-459b-8243-f51e4cf8d7be admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:06.718646 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cb1cf1-216f-459b-8243-f51e4cf8d7be admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:06.723259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4276d5bd-0bfa-4710-82d1-0bba391dc2f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:06.723842 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4276d5bd-0bfa-4710-82d1-0bba391dc2f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:06.724119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4276d5bd-0bfa-4710-82d1-0bba391dc2f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:06.724489 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4276d5bd-0bfa-4710-82d1-0bba391dc2f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:06.725124 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4276d5bd-0bfa-4710-82d1-0bba391dc2f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:06.726726 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4276d5bd-0bfa-4710-82d1-0bba391dc2f8 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:06.736980 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1453/2936] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Sep 28 12:51:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 28 12:51:06.741356 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313cbd6-a9a1-4c03-9c4a-eefb6a03ebcc admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:06.751144 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313cbd6-a9a1-4c03-9c4a-eefb6a03ebcc admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:06.755729 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c311792-987f-40cd-b628-fd054bc7b267 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:06.756399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c311792-987f-40cd-b628-fd054bc7b267 admin admin] REQUEST_METHOD: `PUT` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:06.757184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c311792-987f-40cd-b628-fd054bc7b267 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:06.757457 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c311792-987f-40cd-b628-fd054bc7b267 admin admin] PATH_INFO: `/v3/projects/640d45e3f0f04dd9b3c94d1d5b9dc026/users/41907efd814c4a7fa6f7441f5f05a86e/roles/615e83fab2504b6298db3efaa7c15750` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:06.771818 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c311792-987f-40cd-b628-fd054bc7b267 admin admin] RBAC: Authorizing `identity:create_grant(project_id=640d45e3f0f04dd9b3c94d1d5b9dc026, user_id=41907efd814c4a7fa6f7441f5f05a86e, role_id=615e83fab2504b6298db3efaa7c15750)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:06.775170 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c311792-987f-40cd-b628-fd054bc7b267 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:06.789346 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1484/2937] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Sep 28 12:51:06 2022] PUT /identity/v3/projects/640d45e3f0f04dd9b3c94d1d5b9dc026/users/41907efd814c4a7fa6f7441f5f05a86e/roles/615e83fab2504b6298db3efaa7c15750 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:51:06.799477 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc8ecd7-8458-4929-ace3-e7541e089aff None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:06.799780 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc8ecd7-8458-4929-ace3-e7541e089aff None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:06.800037 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc8ecd7-8458-4929-ace3-e7541e089aff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:06.828480 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-6dc8ecd7-8458-4929-ace3-e7541e089aff None None] MFA Rules not processed for user `41907efd814c4a7fa6f7441f5f05a86e`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:51:06.829751 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-6dc8ecd7-8458-4929-ace3-e7541e089aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:51:06.893771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1454/2938] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:51:06 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:51:06.904952 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c311792-987f-40cd-b628-fd054bc7b267 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:06.906691 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4c311792-987f-40cd-b628-fd054bc7b267 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:51:06.935471 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:51:06.965184 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c311792-987f-40cd-b628-fd054bc7b267 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:06.968794 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e33284-2ea3-4e8e-9093-c79d6bd610f5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:06.969389 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e33284-2ea3-4e8e-9093-c79d6bd610f5 service cinder] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:06.969667 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e33284-2ea3-4e8e-9093-c79d6bd610f5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:06.969936 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e33284-2ea3-4e8e-9093-c79d6bd610f5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:06.978558 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e33284-2ea3-4e8e-9093-c79d6bd610f5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:06.979968 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e33284-2ea3-4e8e-9093-c79d6bd610f5 service cinder] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:06.988751 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1485/2939] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:51:06 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:51:07.014716 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc8ecd7-8458-4929-ace3-e7541e089aff None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:07.075405 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc8ecd7-8458-4929-ace3-e7541e089aff None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:07.079955 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19561c1-078b-4414-934f-889ffcdfa7e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:07.080600 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19561c1-078b-4414-934f-889ffcdfa7e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:07.080881 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19561c1-078b-4414-934f-889ffcdfa7e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:07.081151 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19561c1-078b-4414-934f-889ffcdfa7e9 admin admin] PATH_INFO: `/v3/users/41907efd814c4a7fa6f7441f5f05a86e` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:07.082264 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19561c1-078b-4414-934f-889ffcdfa7e9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=41907efd814c4a7fa6f7441f5f05a86e)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:07.084153 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19561c1-078b-4414-934f-889ffcdfa7e9 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:07.089108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:51:07.130360 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e19561c1-078b-4414-934f-889ffcdfa7e9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '41907efd814c4a7fa6f7441f5f05a86e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:07.131030 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e19561c1-078b-4414-934f-889ffcdfa7e9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '41907efd814c4a7fa6f7441f5f05a86e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:07.131649 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e19561c1-078b-4414-934f-889ffcdfa7e9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '41907efd814c4a7fa6f7441f5f05a86e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:07.138355 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e19561c1-078b-4414-934f-889ffcdfa7e9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '41907efd814c4a7fa6f7441f5f05a86e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:07.151418 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-e19561c1-078b-4414-934f-889ffcdfa7e9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '41907efd814c4a7fa6f7441f5f05a86e'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:07.163843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1455/2940] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:51:07 2022] DELETE /identity/v3/users/41907efd814c4a7fa6f7441f5f05a86e => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:51:07.168385 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc7e235-d89d-4998-8e4c-f7b180622dff None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:07.168675 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc7e235-d89d-4998-8e4c-f7b180622dff None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:07.168976 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc7e235-d89d-4998-8e4c-f7b180622dff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:07.187559 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-4cc7e235-d89d-4998-8e4c-f7b180622dff None None] MFA Rules not processed for user `f82a94e2366b45ff82ac58061224a3a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:51:07.188795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4cc7e235-d89d-4998-8e4c-f7b180622dff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:51:07.245847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1486/2941] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:51:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:51:07.255365 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19561c1-078b-4414-934f-889ffcdfa7e9 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:07.256843 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e19561c1-078b-4414-934f-889ffcdfa7e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:51:07.312434 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19561c1-078b-4414-934f-889ffcdfa7e9 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:07.316056 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3d444b-2395-4147-8536-ab42f8d278a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:07.316672 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3d444b-2395-4147-8536-ab42f8d278a7 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:07.316952 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3d444b-2395-4147-8536-ab42f8d278a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:07.317221 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3d444b-2395-4147-8536-ab42f8d278a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:07.326038 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3d444b-2395-4147-8536-ab42f8d278a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:07.327460 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3d444b-2395-4147-8536-ab42f8d278a7 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:07.337213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1456/2942] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:51:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:51:07.602398 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc7e235-d89d-4998-8e4c-f7b180622dff None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:51:07.603850 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-4cc7e235-d89d-4998-8e4c-f7b180622dff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:51:07.617520 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc7e235-d89d-4998-8e4c-f7b180622dff None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:51:07.622256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae2efb5-a059-455a-9650-a09e43ab614c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:51:07.622878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae2efb5-a059-455a-9650-a09e43ab614c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:51:07.623151 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae2efb5-a059-455a-9650-a09e43ab614c admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:51:07.623424 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae2efb5-a059-455a-9650-a09e43ab614c admin admin] PATH_INFO: `/v3/projects/640d45e3f0f04dd9b3c94d1d5b9dc026` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:51:07.624567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae2efb5-a059-455a-9650-a09e43ab614c admin admin] RBAC: Authorizing `identity:delete_project(project_id=640d45e3f0f04dd9b3c94d1d5b9dc026)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:51:07.626547 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae2efb5-a059-455a-9650-a09e43ab614c admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:51:07.672629 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-aae2efb5-a059-455a-9650-a09e43ab614c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '640d45e3f0f04dd9b3c94d1d5b9dc026', 'request_id': 'req-aae2efb5-a059-455a-9650-a09e43ab614c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:07.678121 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-aae2efb5-a059-455a-9650-a09e43ab614c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '640d45e3f0f04dd9b3c94d1d5b9dc026', 'request_id': 'req-aae2efb5-a059-455a-9650-a09e43ab614c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:07.685154 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-aae2efb5-a059-455a-9650-a09e43ab614c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '640d45e3f0f04dd9b3c94d1d5b9dc026', 'request_id': 'req-aae2efb5-a059-455a-9650-a09e43ab614c'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:07.697126 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-aae2efb5-a059-455a-9650-a09e43ab614c admin admin] The token cache is being invalidate because project 640d45e3f0f04dd9b3c94d1d5b9dc026 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:51:07.697436 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-aae2efb5-a059-455a-9650-a09e43ab614c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:07.698073 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-aae2efb5-a059-455a-9650-a09e43ab614c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:51:07.699646 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1487/2943] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:51:07 2022] DELETE /identity/v3/projects/640d45e3f0f04dd9b3c94d1d5b9dc026 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:51:17.332525 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=50,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:51:17.693145 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=69,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:53:02.675148 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c2dcae-78af-4a41-b03f-869424ef4d2e None None] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:53:02.675752 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c2dcae-78af-4a41-b03f-869424ef4d2e None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:53:02.675752 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c2dcae-78af-4a41-b03f-869424ef4d2e None None] PATH_INFO: `/` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:53:02.677075 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1457/2944] 10.222.0.26 () {54 vars in 921 bytes} [Wed Sep 28 12:53:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:53:02.683355 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2646fb07-77dc-4bc1-8e24-2387954acabb None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:53:02.683842 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2646fb07-77dc-4bc1-8e24-2387954acabb None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:53:02.683949 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2646fb07-77dc-4bc1-8e24-2387954acabb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:53:02.733568 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-2646fb07-77dc-4bc1-8e24-2387954acabb None None] MFA Rules not processed for user `0e429f061bd04c28b0c60c593b64fb8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:53:02.734808 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2646fb07-77dc-4bc1-8e24-2387954acabb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:53:02.790942 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1488/2945] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Sep 28 12:53:02 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:53:02.807119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c2dcae-78af-4a41-b03f-869424ef4d2e None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:53:02.808865 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-c2c2dcae-78af-4a41-b03f-869424ef4d2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:53:02.822460 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c2dcae-78af-4a41-b03f-869424ef4d2e None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:53:02.826025 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a401f516-6627-434f-8c0e-1d4c50a4f2d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:53:02.826611 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a401f516-6627-434f-8c0e-1d4c50a4f2d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:53:02.826890 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a401f516-6627-434f-8c0e-1d4c50a4f2d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:53:02.827166 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a401f516-6627-434f-8c0e-1d4c50a4f2d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:53:02.834201 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a401f516-6627-434f-8c0e-1d4c50a4f2d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:53:02.835623 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a401f516-6627-434f-8c0e-1d4c50a4f2d2 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:53:02.852587 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1458/2946] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:53:02 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:53:03.946546 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0436cdeb-65ca-41d3-af20-f875791feafe None None] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:53:03.947092 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0436cdeb-65ca-41d3-af20-f875791feafe None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:53:03.947092 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0436cdeb-65ca-41d3-af20-f875791feafe None None] PATH_INFO: `/` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:53:03.948743 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1489/2947] 10.222.0.26 () {54 vars in 921 bytes} [Wed Sep 28 12:53:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 28 12:53:03.953758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9ff59c-50a3-4425-8fd6-92f4c365ec10 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:53:03.954247 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9ff59c-50a3-4425-8fd6-92f4c365ec10 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:53:03.954331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9ff59c-50a3-4425-8fd6-92f4c365ec10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:53:04.001078 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-3b9ff59c-50a3-4425-8fd6-92f4c365ec10 None None] MFA Rules not processed for user `8bf969348e7a4158acf8fb789430b0e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:53:04.002347 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-3b9ff59c-50a3-4425-8fd6-92f4c365ec10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:53:04.073528 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1459/2948] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Sep 28 12:53:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:53:04.084779 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0436cdeb-65ca-41d3-af20-f875791feafe None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:53:04.096278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0436cdeb-65ca-41d3-af20-f875791feafe None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:53:04.099827 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96204179-bc77-4366-b51f-c4980dd4b9cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:53:04.100443 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96204179-bc77-4366-b51f-c4980dd4b9cd service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:53:04.100722 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96204179-bc77-4366-b51f-c4980dd4b9cd service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:53:04.100994 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96204179-bc77-4366-b51f-c4980dd4b9cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:53:04.111852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96204179-bc77-4366-b51f-c4980dd4b9cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:53:04.113319 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96204179-bc77-4366-b51f-c4980dd4b9cd service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:53:04.124054 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1490/2949] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:53:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:53:14.063533 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=43 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:53:14.119444 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=39 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:53:17.929473 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9ff59c-50a3-4425-8fd6-92f4c365ec10 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:53:17.939371 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9ff59c-50a3-4425-8fd6-92f4c365ec10 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:53:17.943137 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a163e01b-08ff-408e-96cb-76507f3cbff3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:53:17.943757 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a163e01b-08ff-408e-96cb-76507f3cbff3 service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:53:17.944042 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a163e01b-08ff-408e-96cb-76507f3cbff3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:53:17.944339 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a163e01b-08ff-408e-96cb-76507f3cbff3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:53:17.946819 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a163e01b-08ff-408e-96cb-76507f3cbff3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:53:17.961089 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a163e01b-08ff-408e-96cb-76507f3cbff3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:53:17.963118 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a163e01b-08ff-408e-96cb-76507f3cbff3 service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:53:17.977754 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1460/2950] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:53:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:53:19.284245 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96204179-bc77-4366-b51f-c4980dd4b9cd service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:53:19.285954 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-96204179-bc77-4366-b51f-c4980dd4b9cd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:53:19.298633 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96204179-bc77-4366-b51f-c4980dd4b9cd service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:53:19.302418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce85f08-561e-48cb-b891-ff94f917432a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:53:19.303028 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce85f08-561e-48cb-b891-ff94f917432a service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:53:19.303313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce85f08-561e-48cb-b891-ff94f917432a service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:53:19.303589 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce85f08-561e-48cb-b891-ff94f917432a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:53:19.305422 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2ce85f08-561e-48cb-b891-ff94f917432a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:53:19.319641 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce85f08-561e-48cb-b891-ff94f917432a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:53:19.321631 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce85f08-561e-48cb-b891-ff94f917432a service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:53:19.333361 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1491/2951] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:53:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:53:20.493350 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7497e1-bb00-4310-8ebc-6bdd5a513541 None None] REQUEST_METHOD: `POST` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:53:20.493880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7497e1-bb00-4310-8ebc-6bdd5a513541 None None] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:53:20.493880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7497e1-bb00-4310-8ebc-6bdd5a513541 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:53:20.513310 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.auth.core [None req-fa7497e1-bb00-4310-8ebc-6bdd5a513541 None None] MFA Rules not processed for user `4d70d61439324d4b9c3b3513799b896c`. Rule list: `[]` (Enabled: `True`). {{(pid=81592) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:53:20.514534 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-fa7497e1-bb00-4310-8ebc-6bdd5a513541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:53:20.580497 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1461/2952] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:53:20 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:53:29.327987 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=6 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:53:30.568929 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=31 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:54:18.682173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce85f08-561e-48cb-b891-ff94f917432a service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:54:18.683583 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-2ce85f08-561e-48cb-b891-ff94f917432a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:54:18.791256 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce85f08-561e-48cb-b891-ff94f917432a service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:54:18.796384 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7c9fbc-e560-4366-9be5-0a68121430ff service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:54:18.796988 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7c9fbc-e560-4366-9be5-0a68121430ff service placement] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:54:18.797273 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7c9fbc-e560-4366-9be5-0a68121430ff service placement] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:54:18.797549 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7c9fbc-e560-4366-9be5-0a68121430ff service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:54:18.799303 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-bf7c9fbc-e560-4366-9be5-0a68121430ff service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:54:18.811213 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7c9fbc-e560-4366-9be5-0a68121430ff service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:54:18.812834 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7c9fbc-e560-4366-9be5-0a68121430ff service placement] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:54:18.825719 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1492/2953] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:54:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:54:28.819269 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=48 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:54:56.950061 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7497e1-bb00-4310-8ebc-6bdd5a513541 None None] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:54:56.959590 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7497e1-bb00-4310-8ebc-6bdd5a513541 None None] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:54:56.963107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e15de4-d773-4420-a5f6-0378feb5491e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:54:56.963692 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e15de4-d773-4420-a5f6-0378feb5491e service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:54:56.963967 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e15de4-d773-4420-a5f6-0378feb5491e service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:54:56.964235 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e15de4-d773-4420-a5f6-0378feb5491e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:54:56.966047 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-76e15de4-d773-4420-a5f6-0378feb5491e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:54:57.026029 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e15de4-d773-4420-a5f6-0378feb5491e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:54:57.027493 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e15de4-d773-4420-a5f6-0378feb5491e service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:54:57.036771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1462/2954] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:54:56 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:55:01.971996 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7c9fbc-e560-4366-9be5-0a68121430ff service placement] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:55:01.981955 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7c9fbc-e560-4366-9be5-0a68121430ff service placement] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:55:01.985750 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce6b4de-34a0-4be1-bd80-a28c632769f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:55:01.986370 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce6b4de-34a0-4be1-bd80-a28c632769f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:55:01.986648 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce6b4de-34a0-4be1-bd80-a28c632769f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:55:01.986919 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce6b4de-34a0-4be1-bd80-a28c632769f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:55:01.988660 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-9ce6b4de-34a0-4be1-bd80-a28c632769f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:55:02.054627 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce6b4de-34a0-4be1-bd80-a28c632769f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:55:02.056255 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce6b4de-34a0-4be1-bd80-a28c632769f9 service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:55:02.066492 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1493/2955] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:55:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:55:04.883450 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e15de4-d773-4420-a5f6-0378feb5491e service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:55:04.910308 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e15de4-d773-4420-a5f6-0378feb5491e service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:55:04.916555 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d368df9d-7b74-4773-8a50-3518e0023c82 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:55:04.917694 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d368df9d-7b74-4773-8a50-3518e0023c82 service placement] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:55:04.918188 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d368df9d-7b74-4773-8a50-3518e0023c82 service placement] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:55:04.918681 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d368df9d-7b74-4773-8a50-3518e0023c82 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:55:04.921304 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-d368df9d-7b74-4773-8a50-3518e0023c82 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:55:04.936736 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d368df9d-7b74-4773-8a50-3518e0023c82 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:55:04.939483 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d368df9d-7b74-4773-8a50-3518e0023c82 service placement] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:55:04.956228 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1463/2956] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:55:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:55:08.089392 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce6b4de-34a0-4be1-bd80-a28c632769f9 service neutron] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:55:08.090838 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-9ce6b4de-34a0-4be1-bd80-a28c632769f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:55:08.103835 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce6b4de-34a0-4be1-bd80-a28c632769f9 service neutron] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:55:08.108362 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070a4d20-4f1d-4d35-9788-fd87d6f877bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:55:08.108981 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a4d20-4f1d-4d35-9788-fd87d6f877bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:55:08.109259 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a4d20-4f1d-4d35-9788-fd87d6f877bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:55:08.109526 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a4d20-4f1d-4d35-9788-fd87d6f877bb admin admin] PATH_INFO: `/v3/users/b287a26201c142f983d844c07df169e9` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:55:08.110721 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070a4d20-4f1d-4d35-9788-fd87d6f877bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=b287a26201c142f983d844c07df169e9)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:55:08.112653 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070a4d20-4f1d-4d35-9788-fd87d6f877bb admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:55:08.158316 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-070a4d20-4f1d-4d35-9788-fd87d6f877bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b287a26201c142f983d844c07df169e9'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.158916 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-070a4d20-4f1d-4d35-9788-fd87d6f877bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b287a26201c142f983d844c07df169e9'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.166278 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-070a4d20-4f1d-4d35-9788-fd87d6f877bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b287a26201c142f983d844c07df169e9'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.166900 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-070a4d20-4f1d-4d35-9788-fd87d6f877bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b287a26201c142f983d844c07df169e9'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.179057 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-070a4d20-4f1d-4d35-9788-fd87d6f877bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b287a26201c142f983d844c07df169e9'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.192223 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1494/2957] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:55:08 2022] DELETE /identity/v3/users/b287a26201c142f983d844c07df169e9 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:55:08.360734 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d368df9d-7b74-4773-8a50-3518e0023c82 service placement] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:55:08.372268 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d368df9d-7b74-4773-8a50-3518e0023c82 service placement] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:55:08.428997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:55:08.429616 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:55:08.429891 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:55:08.430158 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] PATH_INFO: `/v3/projects/325a146c46e54eec898106e7b88a86b6` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:55:08.431254 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] RBAC: Authorizing `identity:delete_project(project_id=325a146c46e54eec898106e7b88a86b6)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:55:08.433221 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:55:08.480004 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '325a146c46e54eec898106e7b88a86b6', 'request_id': 'req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.498518 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '325a146c46e54eec898106e7b88a86b6', 'request_id': 'req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.506171 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '325a146c46e54eec898106e7b88a86b6', 'request_id': 'req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.520036 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] The token cache is being invalidate because project 325a146c46e54eec898106e7b88a86b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:55:08.520358 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.521015 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.522506 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1464/2958] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:55:08 2022] DELETE /identity/v3/projects/325a146c46e54eec898106e7b88a86b6 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:55:08.527124 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070a4d20-4f1d-4d35-9788-fd87d6f877bb admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:55:08.528720 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-070a4d20-4f1d-4d35-9788-fd87d6f877bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:55:08.542332 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070a4d20-4f1d-4d35-9788-fd87d6f877bb admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:55:08.599878 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f302f44-2ec8-4759-9aa2-e72a9d214071 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:55:08.600532 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f302f44-2ec8-4759-9aa2-e72a9d214071 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:55:08.600858 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f302f44-2ec8-4759-9aa2-e72a9d214071 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:55:08.601136 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f302f44-2ec8-4759-9aa2-e72a9d214071 admin admin] PATH_INFO: `/v3/users/98f92541a37e478db5d35ac2560301af` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:55:08.602313 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f302f44-2ec8-4759-9aa2-e72a9d214071 admin admin] RBAC: Authorizing `identity:delete_user(user_id=98f92541a37e478db5d35ac2560301af)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:55:08.604221 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f302f44-2ec8-4759-9aa2-e72a9d214071 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:55:08.609671 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:55:08.650924 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1f302f44-2ec8-4759-9aa2-e72a9d214071 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '98f92541a37e478db5d35ac2560301af'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.651576 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1f302f44-2ec8-4759-9aa2-e72a9d214071 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '98f92541a37e478db5d35ac2560301af'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.658517 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1f302f44-2ec8-4759-9aa2-e72a9d214071 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '98f92541a37e478db5d35ac2560301af'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.659153 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1f302f44-2ec8-4759-9aa2-e72a9d214071 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '98f92541a37e478db5d35ac2560301af'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.670947 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-1f302f44-2ec8-4759-9aa2-e72a9d214071 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '98f92541a37e478db5d35ac2560301af'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.683000 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1495/2959] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:55:08 2022] DELETE /identity/v3/users/98f92541a37e478db5d35ac2560301af => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:55:08.852804 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:55:08.854343 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:55:08.868119 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b13f3ce-b213-4850-a60b-6c0dc0c3d92f admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:55:08.880259 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:55:08.932916 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:55:08.933560 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:55:08.933837 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:55:08.934107 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] PATH_INFO: `/v3/projects/27f9bf030d2741eb852052dc2c16a381` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:55:08.935258 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=27f9bf030d2741eb852052dc2c16a381)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:55:08.937219 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:55:08.986101 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27f9bf030d2741eb852052dc2c16a381', 'request_id': 'req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:08.992094 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27f9bf030d2741eb852052dc2c16a381', 'request_id': 'req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:09.000340 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27f9bf030d2741eb852052dc2c16a381', 'request_id': 'req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:09.012328 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] The token cache is being invalidate because project 27f9bf030d2741eb852052dc2c16a381 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81592) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:55:09.012824 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:09.013541 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:55:09.015020 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1465/2960] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:55:08 2022] DELETE /identity/v3/projects/27f9bf030d2741eb852052dc2c16a381 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 28 12:55:18.682136 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=23,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:55:19.008935 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=44,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:55:20.970939 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f302f44-2ec8-4759-9aa2-e72a9d214071 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:55:20.972507 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-1f302f44-2ec8-4759-9aa2-e72a9d214071 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:55:21.028203 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f302f44-2ec8-4759-9aa2-e72a9d214071 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:55:21.074284 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba0b3e4-cb42-4369-b1d7-cd69bdb0f7ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:55:21.074874 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba0b3e4-cb42-4369-b1d7-cd69bdb0f7ae service neutron] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:55:21.075148 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba0b3e4-cb42-4369-b1d7-cd69bdb0f7ae service neutron] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:55:21.075418 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba0b3e4-cb42-4369-b1d7-cd69bdb0f7ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:55:21.077147 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7ba0b3e4-cb42-4369-b1d7-cd69bdb0f7ae service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:55:21.086173 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba0b3e4-cb42-4369-b1d7-cd69bdb0f7ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:55:21.088058 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba0b3e4-cb42-4369-b1d7-cd69bdb0f7ae service neutron] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:55:21.095901 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1496/2961] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:55:20 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:55:31.092299 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=46 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:55:45.639559 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:55:45.640991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:55:45.696651 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db64e41-3b91-4e09-9e1a-9bd344fe61cf admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:55:45.701087 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3615405-f9ab-447b-9dd7-9ffd0d3fdbcb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ba80861b08940d398a61879bb70060a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:55:45.701685 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3615405-f9ab-447b-9dd7-9ffd0d3fdbcb service cinder] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:55:45.701966 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3615405-f9ab-447b-9dd7-9ffd0d3fdbcb service cinder] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:55:45.702236 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3615405-f9ab-447b-9dd7-9ffd0d3fdbcb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:55:45.703956 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f3615405-f9ab-447b-9dd7-9ffd0d3fdbcb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:55:45.760580 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3615405-f9ab-447b-9dd7-9ffd0d3fdbcb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:55:45.762561 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3615405-f9ab-447b-9dd7-9ffd0d3fdbcb service cinder] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:55:45.771809 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1466/2962] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:55:45 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:55:55.767388 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=48 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:55:56.766847 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff339c3-d87d-4be3-8793-b75323c4e19d None None] REQUEST_METHOD: `POST` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:55:56.767336 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff339c3-d87d-4be3-8793-b75323c4e19d None None] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:55:56.767451 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff339c3-d87d-4be3-8793-b75323c4e19d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:55:56.786209 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.auth.core [None req-5ff339c3-d87d-4be3-8793-b75323c4e19d None None] MFA Rules not processed for user `4d70d61439324d4b9c3b3513799b896c`. Rule list: `[]` (Enabled: `True`). {{(pid=81593) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 28 12:55:56.787380 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5ff339c3-d87d-4be3-8793-b75323c4e19d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:55:56.794836 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1497/2963] 10.222.0.26 () {58 vars in 958 bytes} [Wed Sep 28 12:55:56 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 28 12:55:57.131856 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3615405-f9ab-447b-9dd7-9ffd0d3fdbcb service cinder] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:55:57.133279 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-f3615405-f9ab-447b-9dd7-9ffd0d3fdbcb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:55:57.145213 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3615405-f9ab-447b-9dd7-9ffd0d3fdbcb service cinder] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:55:57.148753 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ae559b-724f-40fc-a74b-ebacea4cf66d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:55:57.149338 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ae559b-724f-40fc-a74b-ebacea4cf66d service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:55:57.149617 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ae559b-724f-40fc-a74b-ebacea4cf66d service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:55:57.149889 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ae559b-724f-40fc-a74b-ebacea4cf66d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:55:57.151573 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a3ae559b-724f-40fc-a74b-ebacea4cf66d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:55:57.161316 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ae559b-724f-40fc-a74b-ebacea4cf66d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:55:57.162729 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ae559b-724f-40fc-a74b-ebacea4cf66d service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:55:57.170542 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1467/2964] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:55:57 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:55:58.622909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff339c3-d87d-4be3-8793-b75323c4e19d None None] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:55:58.624396 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-5ff339c3-d87d-4be3-8793-b75323c4e19d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:55:58.693197 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff339c3-d87d-4be3-8793-b75323c4e19d None None] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:55:58.698137 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1f4d39-6a3c-4bcc-b0d2-0c3ce65d61d4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:55:58.698732 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1f4d39-6a3c-4bcc-b0d2-0c3ce65d61d4 service placement] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:55:58.699008 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1f4d39-6a3c-4bcc-b0d2-0c3ce65d61d4 service placement] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:55:58.699276 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1f4d39-6a3c-4bcc-b0d2-0c3ce65d61d4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:55:58.701525 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7f1f4d39-6a3c-4bcc-b0d2-0c3ce65d61d4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:55:58.712659 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1f4d39-6a3c-4bcc-b0d2-0c3ce65d61d4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:55:58.714210 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1f4d39-6a3c-4bcc-b0d2-0c3ce65d61d4 service placement] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:55:58.724506 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1498/2965] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:55:58 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:56:01.077108 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ae559b-724f-40fc-a74b-ebacea4cf66d service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:56:01.079953 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-a3ae559b-724f-40fc-a74b-ebacea4cf66d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:56:01.159361 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ae559b-724f-40fc-a74b-ebacea4cf66d service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:56:01.165176 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8098a9-2526-41f3-92c8-671abfafd9d0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:56:01.165908 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8098a9-2526-41f3-92c8-671abfafd9d0 service nova] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:56:01.166199 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8098a9-2526-41f3-92c8-671abfafd9d0 service nova] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:56:01.166464 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8098a9-2526-41f3-92c8-671abfafd9d0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:56:01.168217 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-dd8098a9-2526-41f3-92c8-671abfafd9d0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:56:01.180456 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8098a9-2526-41f3-92c8-671abfafd9d0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:56:01.182122 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8098a9-2526-41f3-92c8-671abfafd9d0 service nova] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:56:01.194165 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1468/2966] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:56:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:56:08.720359 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=35 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:56:11.189563 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=40 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:56:13.009763 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1f4d39-6a3c-4bcc-b0d2-0c3ce65d61d4 service placement] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:56:13.011252 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7f1f4d39-6a3c-4bcc-b0d2-0c3ce65d61d4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:56:13.030258 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1f4d39-6a3c-4bcc-b0d2-0c3ce65d61d4 service placement] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:56:13.035572 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b328f1f-a93d-4163-80f8-5078685667e1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:56:13.036159 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b328f1f-a93d-4163-80f8-5078685667e1 service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:56:13.036468 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b328f1f-a93d-4163-80f8-5078685667e1 service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:56:13.036737 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b328f1f-a93d-4163-80f8-5078685667e1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:56:13.038440 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-8b328f1f-a93d-4163-80f8-5078685667e1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:56:13.050452 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b328f1f-a93d-4163-80f8-5078685667e1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:56:13.052013 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b328f1f-a93d-4163-80f8-5078685667e1 service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:56:13.062557 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1499/2967] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:56:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:56:16.428886 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8098a9-2526-41f3-92c8-671abfafd9d0 service nova] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:56:16.430293 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-dd8098a9-2526-41f3-92c8-671abfafd9d0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:56:16.441997 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8098a9-2526-41f3-92c8-671abfafd9d0 service nova] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:56:16.445907 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732f5b4a-3735-4a33-8978-866e5019044e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:56:16.446449 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732f5b4a-3735-4a33-8978-866e5019044e service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:56:16.446771 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732f5b4a-3735-4a33-8978-866e5019044e service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:56:16.447045 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732f5b4a-3735-4a33-8978-866e5019044e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:56:16.448781 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-732f5b4a-3735-4a33-8978-866e5019044e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:56:16.513742 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732f5b4a-3735-4a33-8978-866e5019044e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:56:16.515180 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732f5b4a-3735-4a33-8978-866e5019044e service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:56:16.524584 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1469/2968] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:56:16 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:56:18.302438 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b328f1f-a93d-4163-80f8-5078685667e1 service nova] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:56:18.337222 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b328f1f-a93d-4163-80f8-5078685667e1 service nova] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:56:18.343710 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3777715-0805-4198-852a-3fcf460b070a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8bf969348e7a4158acf8fb789430b0e6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:56:18.344399 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3777715-0805-4198-852a-3fcf460b070a service nova] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:56:18.344703 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3777715-0805-4198-852a-3fcf460b070a service nova] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:56:18.344980 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3777715-0805-4198-852a-3fcf460b070a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:56:18.346711 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-d3777715-0805-4198-852a-3fcf460b070a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:56:18.369922 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3777715-0805-4198-852a-3fcf460b070a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:56:18.371521 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3777715-0805-4198-852a-3fcf460b070a service nova] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:56:18.382085 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1500/2969] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:56:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:56:26.520331 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=30 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:56:28.377567 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=27 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:58:21.067053 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732f5b4a-3735-4a33-8978-866e5019044e service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:58:21.094924 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732f5b4a-3735-4a33-8978-866e5019044e service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:58:21.098490 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632f75a8-53cb-4e22-88ff-7a8a4f62d33f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e429f061bd04c28b0c60c593b64fb8e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:58:21.099081 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632f75a8-53cb-4e22-88ff-7a8a4f62d33f service neutron] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:58:21.099368 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632f75a8-53cb-4e22-88ff-7a8a4f62d33f service neutron] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:58:21.099637 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632f75a8-53cb-4e22-88ff-7a8a4f62d33f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:58:21.101372 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-632f75a8-53cb-4e22-88ff-7a8a4f62d33f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:58:21.182572 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632f75a8-53cb-4e22-88ff-7a8a4f62d33f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:58:21.183991 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632f75a8-53cb-4e22-88ff-7a8a4f62d33f service neutron] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:58:21.202508 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1470/2970] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:58:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:58:24.294696 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3777715-0805-4198-852a-3fcf460b070a service nova] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:58:24.307194 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3777715-0805-4198-852a-3fcf460b070a service nova] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:58:24.317131 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7053b4c9-c86e-4782-b501-33e8163c7aec service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:58:24.317885 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7053b4c9-c86e-4782-b501-33e8163c7aec service placement] REQUEST_METHOD: `GET` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:58:24.318573 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7053b4c9-c86e-4782-b501-33e8163c7aec service placement] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:58:24.318843 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7053b4c9-c86e-4782-b501-33e8163c7aec service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:58:24.320740 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-7053b4c9-c86e-4782-b501-33e8163c7aec service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:58:24.335563 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7053b4c9-c86e-4782-b501-33e8163c7aec service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:58:24.338011 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7053b4c9-c86e-4782-b501-33e8163c7aec service placement] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:58:24.353226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1501/2971] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:58:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:58:27.432404 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632f75a8-53cb-4e22-88ff-7a8a4f62d33f service neutron] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:58:27.437434 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-632f75a8-53cb-4e22-88ff-7a8a4f62d33f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:58:27.463592 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632f75a8-53cb-4e22-88ff-7a8a4f62d33f service neutron] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:58:27.471216 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f794bd2-2dfc-4c57-96e5-04fec82892c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:58:27.472553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f794bd2-2dfc-4c57-96e5-04fec82892c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:58:27.473020 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f794bd2-2dfc-4c57-96e5-04fec82892c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:58:27.473341 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f794bd2-2dfc-4c57-96e5-04fec82892c4 admin admin] PATH_INFO: `/v3/users/4d70d61439324d4b9c3b3513799b896c` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:58:27.474594 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f794bd2-2dfc-4c57-96e5-04fec82892c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4d70d61439324d4b9c3b3513799b896c)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:58:27.477295 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f794bd2-2dfc-4c57-96e5-04fec82892c4 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:58:27.535847 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8f794bd2-2dfc-4c57-96e5-04fec82892c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4d70d61439324d4b9c3b3513799b896c'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:27.536638 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8f794bd2-2dfc-4c57-96e5-04fec82892c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4d70d61439324d4b9c3b3513799b896c'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:27.537316 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8f794bd2-2dfc-4c57-96e5-04fec82892c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4d70d61439324d4b9c3b3513799b896c'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:27.547334 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8f794bd2-2dfc-4c57-96e5-04fec82892c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4d70d61439324d4b9c3b3513799b896c'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:27.559303 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-8f794bd2-2dfc-4c57-96e5-04fec82892c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4d70d61439324d4b9c3b3513799b896c'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:27.573817 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1471/2972] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:58:27 2022] DELETE /identity/v3/users/4d70d61439324d4b9c3b3513799b896c => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:58:27.756106 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7053b4c9-c86e-4782-b501-33e8163c7aec service placement] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:58:27.766795 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7053b4c9-c86e-4782-b501-33e8163c7aec service placement] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:58:27.822923 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:58:27.823544 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:58:27.823824 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:58:27.824095 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] PATH_INFO: `/v3/projects/1c8a6694cad9480f86dc4dca113bb762` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:58:27.825236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c8a6694cad9480f86dc4dca113bb762)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:58:27.827129 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:58:27.872622 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c8a6694cad9480f86dc4dca113bb762', 'request_id': 'req-676138d8-786c-462d-9ec9-2bd31d876e93'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:27.878009 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c8a6694cad9480f86dc4dca113bb762', 'request_id': 'req-676138d8-786c-462d-9ec9-2bd31d876e93'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:27.884852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c8a6694cad9480f86dc4dca113bb762', 'request_id': 'req-676138d8-786c-462d-9ec9-2bd31d876e93'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:27.895315 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] The token cache is being invalidate because project 1c8a6694cad9480f86dc4dca113bb762 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:58:27.895604 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:27.896236 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:27.897714 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1502/2973] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:58:27 2022] DELETE /identity/v3/projects/1c8a6694cad9480f86dc4dca113bb762 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:58:27.901961 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f794bd2-2dfc-4c57-96e5-04fec82892c4 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:58:27.903379 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-8f794bd2-2dfc-4c57-96e5-04fec82892c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:58:27.916994 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f794bd2-2dfc-4c57-96e5-04fec82892c4 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:58:27.929081 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:58:27.975713 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91584fce-0e9b-457d-a37b-aa75e60ccb26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:58:27.976250 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91584fce-0e9b-457d-a37b-aa75e60ccb26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:58:27.976553 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91584fce-0e9b-457d-a37b-aa75e60ccb26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:58:27.976822 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91584fce-0e9b-457d-a37b-aa75e60ccb26 admin admin] PATH_INFO: `/v3/users/ca66846a50534121b1f997cdc85859f9` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:58:27.977951 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91584fce-0e9b-457d-a37b-aa75e60ccb26 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca66846a50534121b1f997cdc85859f9)` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:58:27.979851 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91584fce-0e9b-457d-a37b-aa75e60ccb26 admin admin] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:58:28.023487 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-91584fce-0e9b-457d-a37b-aa75e60ccb26 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca66846a50534121b1f997cdc85859f9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:28.024121 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-91584fce-0e9b-457d-a37b-aa75e60ccb26 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca66846a50534121b1f997cdc85859f9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:28.024748 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-91584fce-0e9b-457d-a37b-aa75e60ccb26 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca66846a50534121b1f997cdc85859f9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:28.031598 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-91584fce-0e9b-457d-a37b-aa75e60ccb26 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca66846a50534121b1f997cdc85859f9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:28.043146 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.notifications [None req-91584fce-0e9b-457d-a37b-aa75e60ccb26 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca66846a50534121b1f997cdc85859f9'} {{(pid=81592) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:28.055645 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1472/2974] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Sep 28 12:58:27 2022] DELETE /identity/v3/users/ca66846a50534121b1f997cdc85859f9 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:58:28.221165 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] Authenticating user token {{(pid=81593) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:58:28.222676 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.fernet_utils [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81593) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:58:28.238647 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676138d8-786c-462d-9ec9-2bd31d876e93 admin admin] Validating token access rules against request {{(pid=81593) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:58:28.294030 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b006f9f-2ca8-45df-8c7c-4243d4836275 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f82a94e2366b45ff82ac58061224a3a2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8d6b5c1f82154555b8d48a213cdda1ed', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81593) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:58:28.294652 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b006f9f-2ca8-45df-8c7c-4243d4836275 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:58:28.294929 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b006f9f-2ca8-45df-8c7c-4243d4836275 admin admin] SCRIPT_NAME: `/identity` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:58:28.295198 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b006f9f-2ca8-45df-8c7c-4243d4836275 admin admin] PATH_INFO: `/v3/projects/0fbb37c05a8d4c5bbeb0ba5adda9ac73` {{(pid=81593) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:58:28.296325 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b006f9f-2ca8-45df-8c7c-4243d4836275 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0fbb37c05a8d4c5bbeb0ba5adda9ac73)` {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:58:28.298226 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b006f9f-2ca8-45df-8c7c-4243d4836275 admin admin] RBAC: Authorization granted {{(pid=81593) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:58:28.320796 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:58:28.345928 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0b006f9f-2ca8-45df-8c7c-4243d4836275 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0fbb37c05a8d4c5bbeb0ba5adda9ac73', 'request_id': 'req-0b006f9f-2ca8-45df-8c7c-4243d4836275'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:28.351998 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0b006f9f-2ca8-45df-8c7c-4243d4836275 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0fbb37c05a8d4c5bbeb0ba5adda9ac73', 'request_id': 'req-0b006f9f-2ca8-45df-8c7c-4243d4836275'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:28.359179 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0b006f9f-2ca8-45df-8c7c-4243d4836275 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0fbb37c05a8d4c5bbeb0ba5adda9ac73', 'request_id': 'req-0b006f9f-2ca8-45df-8c7c-4243d4836275'} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:28.370909 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0b006f9f-2ca8-45df-8c7c-4243d4836275 admin admin] The token cache is being invalidate because project 0fbb37c05a8d4c5bbeb0ba5adda9ac73 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81593) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 28 12:58:28.371199 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0b006f9f-2ca8-45df-8c7c-4243d4836275 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:28.371852 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG keystone.notifications [None req-0b006f9f-2ca8-45df-8c7c-4243d4836275 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81593) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 28 12:58:28.373360 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: [pid: 81593|app: 0|req: 1503/2975] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Sep 28 12:58:28 2022] DELETE /identity/v3/projects/0fbb37c05a8d4c5bbeb0ba5adda9ac73 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 28 12:58:38.052090 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=45,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:58:38.367810 dv-cinder-859193-2-iscsi devstack@keystone.service[81593]: DEBUG dbcounter [-] [81593] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81593) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 28 12:59:21.664498 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91584fce-0e9b-457d-a37b-aa75e60ccb26 admin admin] Authenticating user token {{(pid=81592) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 28 12:59:21.665899 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-91584fce-0e9b-457d-a37b-aa75e60ccb26 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:59:21.733065 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91584fce-0e9b-457d-a37b-aa75e60ccb26 admin admin] Validating token access rules against request {{(pid=81592) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 28 12:59:21.789880 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1610b48-55dc-42e7-ab7b-bef88830d665 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3758ea0f26264340ad2f154c0111af21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8695d562e3e24c7c9300cf3a23cd7502', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81592) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 28 12:59:21.790481 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1610b48-55dc-42e7-ab7b-bef88830d665 service placement] REQUEST_METHOD: `GET` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 28 12:59:21.790758 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1610b48-55dc-42e7-ab7b-bef88830d665 service placement] SCRIPT_NAME: `/identity` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 28 12:59:21.791030 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1610b48-55dc-42e7-ab7b-bef88830d665 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81592) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 28 12:59:21.792779 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.fernet_utils [None req-e1610b48-55dc-42e7-ab7b-bef88830d665 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81592) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 28 12:59:21.803830 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1610b48-55dc-42e7-ab7b-bef88830d665 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 28 12:59:21.805429 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1610b48-55dc-42e7-ab7b-bef88830d665 service placement] RBAC: Authorization granted {{(pid=81592) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 28 12:59:21.815149 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: [pid: 81592|app: 0|req: 1473/2976] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Sep 28 12:59:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 28 12:59:31.810876 dv-cinder-859193-2-iscsi devstack@keystone.service[81592]: DEBUG dbcounter [-] [81592] Writing DB stats keystone:SELECT=58 {{(pid=81592) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}